Wednesday, February 24, 2010

Using a Webcam with DirectShowNET and F#

In this post I'm going to show a small F# example of using DirectShowNET to access a webcam and manipulate the image data.

DirectShow


DirectShow is a huge Windows API for video playback and capture. Among many things this API allows flexible access to data captured by video input devices such as a webcam.

DirectShowNET


The DirectShow is COM based API. According to the documentation, it's meant to be used from C++. Luckily there's DirectShowNET which is a very nice library that exposes the DirectShow interfaces to .NET languages .

Accessing the webcam


What I wanted to do for this example is to have access to the data of the image being captured at a given time. The DirectShow API provides the concept of a Sample Grabber which not only gives you access to the captured data, but also allows its modification .

The following example shows the use of a couple of functions defined below:


let device = findCaptureDevice

let mediaControl,filterGraph = createVideoCaptureWithSampleGrabber
device
nullGrabber
None



let form = new Form(Size = new Size(300,300), Visible = true,Text = "Webcam input")


let videoWindow = configureVideoWindow (form.Handle) 300 300 filterGraph

form.Closing.Add (fun _ ->
mediaControl.StopWhenReady()
Marshal.ReleaseComObject(videoWindow) |> ignore
Marshal.ReleaseComObject(mediaControl) |> ignore
Marshal.ReleaseComObject(filterGraph) |> ignore)


mediaControl.Run() |> ignore

Application.Run(form)


Running this program will show the following window:



Notice that we called the createVideoCaptureWithSampleGrabber function using the nullGrabber parameter. This parameter specifies a sample grabber that does nothing with the current frame. Here's the definition:


let nullGrabber =
fun (width,height) ->
{ new ISampleGrabberCB with
member this.SampleCB(sampleTime:double , pSample:IMediaSample )= 0
member this.BufferCB(sampleTime:double , pBuffer:System.IntPtr , bufferLen:int) = 0
}


We can change this sample grabber to something more interesting:


let incrementGrabber =
fun (width,height) ->
{ new ISampleGrabberCB with
member this.SampleCB(sampleTime:double , pSample:IMediaSample )= 0
member this.BufferCB(sampleTime:double , pBuffer:System.IntPtr , bufferLen:int) =
for i = 0 to (bufferLen - 1) do
let c = Marshal.ReadByte(pBuffer,i)
Marshal.WriteByte(pBuffer,i ,if c > byte(150) then byte(255) else c+byte(100))
0 }



We can change the program to use:


let mediaControl,filterGraph = createVideoCaptureWithSampleGrabber
device
incrementGrabber
None


Running the program we can see:



This new sample grabber increments each pixel value by 100 or leaves the maximum value to prevent overflow. As presented here the pixel information is provided as an unmanaged pointer.


Using DirectShowNet



DirectShow also has the concept of filters which are software components that are assembled together to mix various inputs and outputs. For this examples we're going to used only a couple of interfaces. The code for this post is based on the DxLogo sample provided with DirectShowNET.

First we define the module containing these functions:


module LangExplrExperiments.DirectShowCapture

open DirectShowLib
open System.Runtime.InteropServices
open System.Runtime.InteropServices.ComTypes


let private checkResult hresult = DsError.ThrowExceptionForHR( hresult )


The checkResult function is an utility function to check for the HRESULT of some of the calls to COM interfaces. Since DirectShowNET is a thin wrapper for these interfaces we need to use this function to check for errors.

The findCaptureDevice returns the first capture devices detected by DirectShowNET .


let findCaptureDevice =
let devices = DsDevice.GetDevicesOfCat(FilterCategory.VideoInputDevice)
let source : obj ref = ref null
devices.[0].Mon.BindToObject(null,null,ref typeof<IBaseFilter>.GUID,source)
devices.[0]


The following functions define the capture filter and sample grabber. The createVideoCaptureWithSampleGrabber function is the entry point. We can specify a file name as the final parameter to save the captured data to a video file.


let private ConfigureSampleGrabber( sampGrabber:ISampleGrabber,callbackobject:ISampleGrabberCB) =
let media = new AMMediaType()

media.majorType <- MediaType.Video
media.subType <- MediaSubType.RGB24
media.formatType <- FormatType.VideoInfo

sampGrabber.SetMediaType( media ) |> checkResult
DsUtils.FreeAMMediaType(media);

sampGrabber.SetCallback( callbackobject, 1 ) |> checkResult


let getCaptureResolution(capGraph:ICaptureGraphBuilder2 , capFilter:IBaseFilter) =
let o : obj ref = ref null
let media : AMMediaType ref = ref null
let videoControl = capFilter :?> IAMVideoControl

capGraph.FindInterface(new DsGuid( PinCategory.Capture),
new DsGuid( MediaType.Video),
capFilter,
typeof<IAMStreamConfig>.GUID,
o ) |> checkResult

let videoStreamConfig = o.Value :?> IAMStreamConfig;

videoStreamConfig.GetFormat(media) |> checkResult

let v = new VideoInfoHeader()
Marshal.PtrToStructure( media.Value.formatPtr, v )
DsUtils.FreeAMMediaType(media.Value)

v.BmiHeader.Width,v.BmiHeader.Height

let createCaptureFilter (captureDevice:DsDevice)
(sampleGrabberCBCreator: int*int -> ISampleGrabberCB) =
let captureGraphBuilder = box(new CaptureGraphBuilder2()) :?> ICaptureGraphBuilder2
let sampGrabber = box(new SampleGrabber()) :?> ISampleGrabber;
let filterGraph = box(new FilterGraph()) :?> IFilterGraph2
let capFilter: IBaseFilter ref = ref null

captureGraphBuilder.SetFiltergraph(filterGraph) |> checkResult

filterGraph.AddSourceFilterForMoniker(
captureDevice.Mon,
null,
captureDevice.Name,
capFilter) |> checkResult


let resolution = getCaptureResolution(captureGraphBuilder,capFilter.Value)
ConfigureSampleGrabber(sampGrabber, sampleGrabberCBCreator(resolution) )

filterGraph.AddFilter(box(sampGrabber) :?> IBaseFilter , "FSGrabberFilter") |> checkResult


captureGraphBuilder,filterGraph,sampGrabber,capFilter.Value

let getMediaControl (captureGraphBuilder :ICaptureGraphBuilder2)
(sampGrabber: ISampleGrabber)
(capFilter:IBaseFilter)
(filterGraph:IFilterGraph2)
(fileNameOpt:string option)=
let muxFilter: IBaseFilter ref = ref null
let fileWriterFilter : IFileSinkFilter ref = ref null
try
match fileNameOpt with
| Some filename ->
captureGraphBuilder.SetOutputFileName(
MediaSubType.Avi,
filename,
muxFilter,
fileWriterFilter) |> checkResult

| None -> ()

captureGraphBuilder.RenderStream(
new DsGuid( PinCategory.Capture),
new DsGuid( MediaType.Video),
capFilter,
sampGrabber :?> IBaseFilter,
muxFilter.Value) |> checkResult

finally
if fileWriterFilter.Value <> null then
Marshal.ReleaseComObject(fileWriterFilter.Value) |> ignore
if muxFilter.Value <> null then
Marshal.ReleaseComObject(muxFilter.Value) |> ignore

filterGraph :?> IMediaControl


let createVideoCaptureWithSampleGrabber (device:DsDevice)
(sampleGrabberCBCreator: int*int -> ISampleGrabberCB)
(outputFileName: string option) =
let capGraphBuilder,filterGraph,sampGrabber,capFilter = createCaptureFilter device sampleGrabberCBCreator

let mediaControl = getMediaControl capGraphBuilder sampGrabber capFilter filterGraph outputFileName

Marshal.ReleaseComObject capGraphBuilder |> ignore
Marshal.ReleaseComObject capFilter |> ignore
Marshal.ReleaseComObject sampGrabber |> ignore

mediaControl,filterGraph



Also for the creation of the video window the following function is provided:

let configureVideoWindow windowHandle width height (filterGraph:IFilterGraph2) =
let videoWindow = filterGraph :?> IVideoWindow

videoWindow.put_Owner(windowHandle) |> checkResult
videoWindow.put_WindowStyle(WindowStyle.Child ||| WindowStyle.ClipChildren) |> checkResult
videoWindow.SetWindowPosition(0,0,width,height) |> checkResult
videoWindow.put_Visible(OABool.True)|> checkResult

videoWindow


For future posts I'm going to try some experiments with the pixel data provided by the sample grabber.

Monday, February 1, 2010

Using a Webcam with WIA and F#

In this post I'm going to show a small example of taking a picture using a Webcam with Windows Image Adquisition 1.0 . This API seems to have changed in Vista and above, the following code only applies to XP.

First step, create the interop assemblies:


C:\test\> TlbImp c:\WINDOWS\system32\wiascr.dll
Microsoft (R) .NET Framework Type Library to Assembly Converter 3.5.30729.1
Copyright (C) Microsoft Corporation. All rights reserved.

Type library imported to WIALib.dll

C:\test>TlbImp c:\WINDOWS\system32\wiavideo.dll
Microsoft (R) .NET Framework Type Library to Assembly Converter 3.5.30729.1
Copyright (C) Microsoft Corporation. All rights reserved.

Type library imported to WIAVIDEOLib.dll


Now we launch FSI and load the assemblies:


C:\test> fsi.exe

Microsoft F# Interactive, (c) Microsoft Corporation, All Rights Reserved
F# Version 1.9.7.8, compiling for .NET Framework Version v2.0.50727

For help type #help;;

> #r "WIALib.dll";;

--> Referenced 'C:\development\blog\wiafs\WIALib.dll'

> #r "WIAVIDEOLIB.dll";;

--> Referenced 'C:\development\blog\wiafs\WIAVIDEOLIB.dll'



Now we can list the capture devices:


> wia.Devices |> Seq.cast |> Seq.map (fun (x:WIALib.IWiaDeviceInfo) -> x.Name);;

val it : seq<string> = seq ["My iPod"; "Logitech QuickCam Chat"]



Now I'm going to use the QuickCam to take the picture so we get the information for this device (of type 'StreamingVideo'):


> let devInfo = wia.Devices |> Seq.cast |> Seq.filter (fun (x:WIALib.IWiaDeviceInfo) -> x.Type.Contains("StreamingVideo")) |> Seq.head;;

val devInfo : WIALib.IWiaDeviceInfo

> devInfo.Name;;
val it : string = "Logitech QuickCam Chat"


Now we create an instance of the 'WIAVIDEOLib.WiaVideoClass' which will let us use the Webcam.


> let vid = new WIAVIDEOLib.WiaVideoClass();;
Binding session to 'C:\development\blog\wiafs\WIAVIDEOLib.dll'...

val vid : WIAVIDEOLib.WiaVideoClass

> vid.ImagesDirectory <- "c:\\temp";;
val it : unit = ()
> let h = ref (new WIAVIDEOLib._RemotableHandle());;

val h : WIAVIDEOLib._RemotableHandle ref =
{contents = WIAVIDEOLib._RemotableHandle;}

> vid.CreateVideoByWiaDevID(devInfo.Id,h,0,1);;
val it : unit = ()



The 'ImagesDirectory' property specifies the location for the captured images.

Now we can take a picture:


> let outFileName = ref "";;

val outFileName : string ref = {contents = "";}

> vid.TakePicture( outFileName );;
val it : unit = ()
> outFileName;;
val it : string ref = {contents = "c:\temp\Imagen 009.jpg";}


The 'outFileName' parameter gets the name of the saved image in the images directory.