开心

使用 webworker 计算文件 MD5 和获取图片元数据

大文件计算 MD5 如何减少主线程卡顿？结合 File.slice、SparkMD5 和 Web Worker 做分片计算，并用 OffscreenCanvas 与 ImageBitmap 获取图片宽高及颜色信息。

—次点击4分钟阅读

前言

最近做广告平台的需求比较多一些，然后会有一个需求是获取到广告主上传的素材元数据信息，例如图片获取宽高、颜色、MD5 等，视频资源获取视频宽高，视频时长等，这些在后台获取如果通过同步方式就会有一个比较大的问题，读取文件时间不确定，如果文件过大，很可能造成当前 tab 或者浏览器崩溃，所以需要通过 web worker 多线程获取这些元数据信息，本文为web worker 的使用总结

同步获取 MD5

之前一般通过同步方式获取，如下：

javascript
import SparkMD5 from 'spark-md5'
const spark = new SparkMD5.ArrayBuffer() 
export const createMD5 = file => {
    // 验证file
    return new Promise((resolve,reject)=>{
        const fileReader = new FileReader()
         fileReader.readAsArrayBuffer(file)
         fileReader.onerror = e => {
             console.error(e)
             return reject(e)
             }
        fileReader.onload = () => {
            try{
            resolve(spark.hash(dataBuffer)) 
            }catch(e){
                 console.error(e)
             return reject(e)
            }
            
         }
    })
    
}

这种方式如果是大文件或者批量的文件直接丢到内存会造成主线程卡顿等问题，所以我们需要通过分片并使用 web worker 来获取

分布获取 MD5

首先需要通过 File.Slice 将文件分片，并将分片内容丢给 Spark-md5 处理

spark-md5介绍：SparkMD5 is a fast md5 implementation of the MD5 algorithm. This script is based in the JKM md5 library which is the fastest algorithm around. This is most suitable for browser usage, because nodejs version might be faster.

使用 File.Slice 将文件分片处理

javascript
const fileReader = new FileReader()
const blobSlice =
  File.prototype.slice ||
  //@ts-ignore
  File.prototype.mozSlice ||
  //@ts-ignore
  File.prototype.webkitSlice
const _file = file
fileReader.readAsArrayBuffer(blobSlice.call(_file, 0, _file.size))
fileReader.onload = async function (e) {
  try {
    // picRunner = picRunner || new Pica({ idle: 30000 });
    // console.log("picRunner--->", picRunner);
    // const _picaFile = await picRunner.resizeBuffer({
    //   src: new Uint8Array(e.target.result.slice(0, e.target.result.Length))
    // });
  } catch (error) {
    reject(error)
  }
}
fileReader.onerror = function () {
  reject(new Error("read file error"))
}

创建 web worker

代码如下：

typescript

import SparkMD5 from 'spark-md5'

const _self: any = self

interface Data {
  payload: Payload
  id: string
}

export interface ExifType {
  width?: number
  height?: number
  color?: number[]
  blurhash: string
  md5?: string
}


interface Payload {
  threshold?: number
  input?: any
  output?: string
  name?: string
  type?: string
}
const spark = new SparkMD5.ArrayBuffer()

_self.onmessage = async ({ data: { id, payload } }: { data: Data }) => {
  try {
    spark.append(payload.input)

    const md5 = spark.end()
    _self.postMessage({
      id,
      payload: {
        output: {
          md5,
        },
      },
    })
  } catch (e) {
    console.error(e)
    _self.postMessage({
      id,
      err: e.message,
    })
  }
}
typescript
import { nanoid } from 'nanoid'

export function getMd5Sign(worker: Worker, file: File): Promise<string> {
  return new Promise((resolve, reject) => {
    const id = nanoid(10)
    const fileReader = new FileReader()
    const blobSlice =
      File.prototype.slice ||
      //@ts-ignore
      File.prototype.mozSlice ||
      //@ts-ignore
      File.prototype.webkitSlice
    const _file = file
    const timeStart = new Date().getTime()
    fileReader.readAsArrayBuffer(blobSlice.call(_file, 0, _file.size))
    fileReader.onload = async function (e) {
      try {
        // picRunner = picRunner || new Pica({ idle: 30000 });
        // console.log("picRunner--->", picRunner);
        // const _picaFile = await picRunner.resizeBuffer({
        //   src: new Uint8Array(e.target.result.slice(0, e.target.result.Length))
        // });
        worker.postMessage(
          {
            id,
            payload: {
              input: e.target.result,
              finish: true,
              name: file.name,
              type: file.type,
            },
          },
          [e.target.result as ArrayBuffer],
        )
      } catch (error) {
        reject(error)
      }
    }
    fileReader.onerror = function () {
      reject(new Error('read file error'))
    }
    worker.onmessage = function (event) {
      console.log('total time: ', new Date().getTime() - timeStart)
      const { id: _id, err, payload } = event.data
      if (err) {
        reject(new Error(err))
        return
      }
      if (id === _id) {
        resolve(payload.output)
      }
    }
  })
}

调用如下：

typescript
// 使用vite
import Md5Worker from '@/worker/md5.worker?worker';
import { getMd5Sign } from './get-md5-sign';

const { md5 } = await getMd5Sign(md5Worker?.(), file);

console.log('md5-->', md5)

Web Worker中计算图片宽高和颜色

延伸：目前已经实现了分片获取文件的 MD5，但例如图片我们还需要获取图片的宽高和颜色信息，在JavaScript中，虽然我们通常通过创建Canvas并传入图片对象来直接获取图片的宽高和颜色信息，但在Web Worker环境中，由于缺乏document对象，传统的new Image()和document.createElement(‘canvas’)方法不再适用。然而，幸运的是，JavaScript还提供了两个强大的对象——OffscreenCanvas和ImageBitmap，它们同样在Web Worker中得到支持，使得在无文档对象的环境中操作图片成为可能。

OffscreenCanvas 是 HTML5 Canvas API 的一种扩展，它允许开发者在独立的画布上进行绘图操作，而不是直接在主屏幕画布上操作。这种方法的优点在于可以显著提升性能，尤其是在需要频繁重绘或动态更新的场景中

ImageBitmap 是一种用于Web上的图像处理技术，旨在提高在Canvas中绘制图像的性能。它表示一个位图图像，该图像可以绘制到Canvas上，同时具有低延迟的特性。与传统的ImageData对象相比，ImageBitmap的主要区别在于它可以直接由GPU处理和绘制，而不需要经过CPU的额外操作，从而提高了性能

代码实现：

typescript
function createPixelArray(imgData, pixelCount, quality) {
  const pixels = imgData
  const pixelArray = []

  for (let i = 0, offset, r, g, b, a; i < pixelCount; i = i + quality) {
    offset = i * 4
    r = pixels[offset + 0]
    g = pixels[offset + 1]
    b = pixels[offset + 2]
    a = pixels[offset + 3]

    // If pixel is mostly opaque and not white
    if (typeof a === 'undefined' || a >= 125) {
      if (!(r > 250 && g > 250 && b > 250)) {
        pixelArray.push([r, g, b])
      }
    }
  }
  return pixelArray
}

function validateOptions(options) {
  let { colorCount, quality } = options

  if (typeof colorCount === 'undefined' || !Number.isInteger(colorCount)) {
    colorCount = 10
  } else if (colorCount === 1) {
    throw new Error(
      'colorCount should be between 2 and 20. To get one color, call getColor() instead of getPalette()',
    )
  } else {
    colorCount = Math.max(colorCount, 2)
    colorCount = Math.min(colorCount, 20)
  }

  if (
    typeof quality === 'undefined' ||
    !Number.isInteger(quality) ||
    quality < 1
  ) {
    quality = 10
  }

  return {
    colorCount,
    quality,
  }
}

export { createPixelArray, validateOptions }

上面文件为图片颜色的一些支持方法

typescript
export interface ExifType {
  width?: number
  height?: number
  color?: number[]
  blurhash: string
  md5?: string
}

export const getImageData = (image: any) => {
  const canvas = new OffscreenCanvas(100, 1)
  canvas.width = image.width
  canvas.height = image.height
  const context = canvas.getContext('2d')
  if (!context) {
    return null
  }
  context.drawImage(image, 0, 0, image.width, image.height)
  return context.getImageData(0, 0, image.width, image.height)
}
export function getExifInfo(blob: Blob): Promise<ExifType> {
  return new Promise(async (resolve, reject) => {
    try {
      const image = await createImageBitmap(blob)
      const imageData = getImageData(image)
      if (!imageData) {
        reject(new Error('获取图片信息错误'))
        return
      }
      const blurhash = encode(
        imageData.data,
        imageData.width,
        imageData.height,
        4,
        4,
      )
      const pixelCount = image.width * image.height

      const pixelArray = createPixelArray(imageData.data, pixelCount, 10)
      const cmap = quantize(pixelArray, 5)
      const palette = cmap ? cmap.palette() : null
      const color = palette?.[0]
      resolve({
        blurhash,
        color,
        width: imageData.width,
        height: imageData.height,
      })
    } catch (error) {
      reject(error)
    }
  })
}

然后修改下 Web Worker 方法

typescript
import quantize from '@lokesh.dhakar/quantize/dist/index.mjs'
import { encode } from 'blurhash'
import SparkMD5 from 'spark-md5'

import { createPixelArray } from '../color'

const _self: any = self

interface Data {
  payload: Payload
  id: string
}

export interface ExifType {
  width?: number
  height?: number
  color?: number[]
  blurhash: string
  md5?: string
}

export const getImageData = (image: any) => {
  const canvas = new OffscreenCanvas(100, 1)
  canvas.width = image.width
  canvas.height = image.height
  const context = canvas.getContext('2d')
  if (!context) {
    return null
  }
  context.drawImage(image, 0, 0, image.width, image.height)
  return context.getImageData(0, 0, image.width, image.height)
}
export function getExifInfo(blob: Blob): Promise<ExifType> {
  return new Promise(async (resolve, reject) => {
    try {
      const image = await createImageBitmap(blob)
      const imageData = getImageData(image)
      if (!imageData) {
        reject(new Error('获取图片信息错误'))
        return
      }
      const blurhash = encode(
        imageData.data,
        imageData.width,
        imageData.height,
        4,
        4,
      )
      const pixelCount = image.width * image.height

      const pixelArray = createPixelArray(imageData.data, pixelCount, 10)
      const cmap = quantize(pixelArray, 5)
      const palette = cmap ? cmap.palette() : null
      const color = palette?.[0]
      resolve({
        blurhash,
        color,
        width: imageData.width,
        height: imageData.height,
      })
    } catch (error) {
      reject(error)
    }
  })
}
interface Payload {
  threshold?: number
  input?: any
  output?: string
  name?: string
  type?: string
}
const spark = new SparkMD5.ArrayBuffer()

_self.onmessage = async ({ data: { id, payload } }: { data: Data }) => {
  try {
    spark.append(payload.input)
    // const image = await loadImage(payload.input as string)
    let exitInfo = {}
    if (payload.type?.startsWith('image')) {
      const { blurhash, width, height, color } = await getExifInfo(
        new Blob([payload.input], { type: payload.type }),
      )
      exitInfo = {
        blurhash,
        width,
        height,
        color,
      }
    }

    const md5 = spark.end()
    _self.postMessage({
      id,
      payload: {
        output: {
          ...exitInfo,
          md5,
        },
      },
    })
  } catch (e) {
    console.error(e)
    _self.postMessage({
      id,
      err: e.message,
    })
  }
}

当前我们就完成了通过 Web Worker 计算文件的 MD5，获取图片的宽高等信息