| |
| (41 intermediate revisions by one user not shown) |
| Line 13: | Line 13: |
| ? | * We will read ARGB values directly from the camera, and see how this values can be efficiently packed and unpacked. | ? | * We will read ARGB values directly from the camera, and see how this values can be efficiently packed and unpacked. |
| ? | * We will discover a faster solution (reading values from the YCbCr buffer). | ? | * We will discover a faster solution (reading values from the YCbCr buffer). |
| ? | * We will discuss how to make this processing even better (to use effects on a smaller resolution). | + | * We will discuss how to make this processing even better (to work with a smaller resolution). |
| ? | * We will see an idea how to make an interesting 8-bit style photo app based on these methods. | ? | * We will see an idea how to make an interesting 8-bit style photo app based on these methods. |
| ? | | ? | |
| Line 21: | Line 21: |
| ? | * We will see how to use this approach on WP8 thanks to MonoGame Framework | ? | * We will see how to use this approach on WP8 thanks to MonoGame Framework |
| ? | * We will compare all methods and measure the rendering time | ? | * We will compare all methods and measure the rendering time |
| ? | | ? |
| ? | | ? | |
| ? | == VideoBrush component == | ? | == VideoBrush component == |
| Line 52: | Line 51: |
| ? | </code> | ? | </code> |
| ? | | ? | |
| ? | Do not forget to add an ID_CAP_ISV_CAMERA capability. This is a very simple solution. The stream from the camera is quite distorted and wrongly rotated, but you can fix this by setting a RelativeTransform on the component. But what is a pity, you can't modify this stream in any way. You can't apply any filter or effect on the VideoBrush object, you don't have an access to the raw data of the pixels. | + | Do not forget to add an ID_CAP_ISV_CAMERA capability. This is a very simple solution. The stream from the camera is quite distorted and wrongly rotated, but you can fix this by setting a RelativeTransform on the Rectangle. But what is a real pity, you can't modify this stream in any way. You can't apply any filter or effect on the VideoBrush object, you don't have an access to the raw data of the pixels. |
| ? | | ? | |
| ? | == Reading from the ARGB buffer == | + | == Reading from the buffer == |
| ? | | ? | |
| ? | We will use a different approach. As you can discover, there are some GetPreviewBuffer methods on the PhotoCamera object. This is the way how we can get the raw data from the camera. But we would like also to display this data somewhere, in some fast and efficient way. WritableBitmap component from the Silverlight is very slow, writing data pixel by pixel using this component can take a hundreds of milliseconds. We will need to find another solution. | ? | We will use a different approach. As you can discover, there are some GetPreviewBuffer methods on the PhotoCamera object. This is the way how we can get the raw data from the camera. But we would like also to display this data somewhere, in some fast and efficient way. WritableBitmap component from the Silverlight is very slow, writing data pixel by pixel using this component can take a hundreds of milliseconds. We will need to find another solution. |
| ? | | ? | |
| ? | We will create a new combined Silverlight/XNA project for Windows Phone 7 (we will talk about WP8 later). We would like to combine the ease of XAML and Silverlight with the power of the XNA into one solution. We will use a similar approach to one described in the article [[Use Camera with XNA]]. | + | We will create a new combined Silverlight/XNA project for Windows Phone 7 (we will talk about WP8 later). We would like to combine the ease of XAML and Silverlight with the power of the XNA in one solution. We will use a very similar approach to one described in the article [[Use Camera with XNA]]. |
| ? | | ? | |
| ? | You need to have installed a Windows Phone SDK 7.1. Open your Visual Studio 2010 (Express or the Professional/Ultimate) and create a new Windows Phone Silverlight and XNA Application project. | + | You need to have installed a [http://dev.windowsphone.com/en-us/downloadsdk Windows Phone SDK 7.1]. Open your Visual Studio 2010 (Express or the Professional/Ultimate) and create a new Windows Phone Silverlight and XNA Application project. |
| ? | | ? | |
| ? | [[File:new_xnasilverlight_project.png]] | ? | [[File:new_xnasilverlight_project.png]] |
| Line 70: | Line 69: |
| ? | </code> | ? | </code> |
| ? | | ? | |
| ? | We will create a helper object now. Lets create a new class CamReader and place here these declarations: | + | We will create a helper object now. Lets create a new class '''CamReader.cs''' and place into it these declarations: |
| ? | | ? | |
| ? | <code csharp> | ? | <code csharp> |
| Line 78: | Line 77: |
| ? | </code> | ? | </code> |
| ? | | ? | |
| ? | As you can see, there is some preview buffer and the output buffer. The preview buffer will contain the raw values taken from the camera and will be 640x480 pixels big. Applying effects on this large resolution would be very resource intensive and slow. We do not need such a big resolution for our live preview. Our output buffer will be smaller, for example only 240x240 px. | + | As you can see, there is some preview buffer and the output buffer. The preview buffer will contain the raw values from the camera and will be usually 640x480 pixels big. Applying effects on this large resolution would be very resource intensive and slow. We do not need such a big resolution for the live preview. Our output buffer will be smaller, for example only 240x240 px. All computations and filter calculations will be executed on this smaller resolution. We can scale up this image later, it will be a little blurred, but it does not matter much for us. |
| ? | + | ? |
| ? | + | This is how the CamReader.StartCamera method can look like: |
| ? | + | ? |
| ? | + | <code csharp> |
| ? | + | public void StartCamera(Dispatcher dispatcher, Canvas fakeCamCanvas, Point outputSize) |
| ? | + | { |
| ? | + | ? ? this.outputSize = outputSize; |
| ? | + | ? ? ? ? ? ? |
| ? | + | ? ? if (camera != null) |
| ? | + | ? ? ? ? StopCamera(fakeCamCanvas); |
| ? | + | ? |
| ? | + | ? ? // Create a PhotoCamera instance |
| ? | + | ? ? if (PhotoCamera.IsCameraTypeSupported(CameraType.Primary)) |
| ? | + | ? ? ? ? camera = new PhotoCamera(CameraType.Primary); |
| ? | + | ? ? else if (PhotoCamera.IsCameraTypeSupported(CameraType.FrontFacing)) |
| ? | + | ? ? ? ? camera = new PhotoCamera(CameraType.FrontFacing); |
| ? | + | ? ? else |
| ? | + | ? ? { |
| ? | + | ? ? ? ? System.Windows.MessageBox.Show("Cannot find a camera on this device"); |
| ? | + | ? ? ? ? return; |
| ? | + | ? ? } |
| ? | + | ? |
| ? | + | ? ? // Wait camera initialization before create the buffer image |
| ? | + | ? ? camera.Initialized += (a, b) => |
| ? | + | ? ? { |
| ? | + | ? ? ? ? // Move to UI thread |
| ? | + | ? ? ? ? dispatcher.BeginInvoke(() => |
| ? | + | ? ? ? ? { |
| ? | + | ? ? ? ? ? ? outputBuffer = new int[outputSize.X * outputSize.Y]; |
| ? | + | ? ? ? ? ? ? previewSize = new Point((int)camera.PreviewResolution.Width, (int)camera.PreviewResolution.Height); |
| ? | + | ? ? ? ? ? ? previewBuffer = new int[previewSize.X * previewSize.Y]; |
| ? | + | ? ? ? ? } |
| ? | + | ? ? ? ? ); |
| ? | + | ? ? }; |
| ? | + | ? |
| ? | + | ? ? // Initialize camera |
| ? | + | ? ? // - we need to initialize VideoBrush to given Canvas, otherwise the stream wouldn't be loaded |
| ? | + | ? ? var brush = new VideoBrush(); |
| ? | + | ? ? brush.SetSource(camera); |
| ? | + | ? ? fakeCamCanvas.Background = brush; |
| ? | + | } |
| ? | + | </code> |
| ? | + | ? |
| ? | + | === Reading ARGB data === |
| ? | + | ? |
| ? | + | Now we can design a method for reading ARGB data from the camera: |
| ? | + | ? |
| ? | + | <code csharp> |
| ? | + | public int[] GetBufferFromARGB() |
| ? | + | { |
| ? | + | ? ? if (camera != null && previewBuffer != null) |
| ? | + | ? ? { |
| ? | + | ? ? ? ? // Get preview Image buffer (640x480) |
| ? | + | ? ? ? ? try { camera.GetPreviewBufferArgb32(previewBuffer); } |
| ? | + | ? ? ? ? catch { return null; } |
| ? | + | ? |
| ? | + | ? ? ? ? // Select outputBuffer pixels (outputSize = smaller than 640x480 preview pixels) |
| ? | + | ? ? ? ? CopyValuesToOutputBuffer(); |
| ? | + | ? |
| ? | + | ? ? ? ? // Swap Red and Blue channel (Silverlight -> XNA's texture) |
| ? | + | ? ? ? ? SwapRedBlueChannel(outputBuffer);? ? ? ? ? ? ? ? |
| ? | + | ? ? ? ? return outputBuffer; |
| ? | + | ? ? } |
| ? | + | ? ? return null; |
| ? | + | } |
| ? | + | </code> |
| ? | + | ? |
| ? | + | This method retrieves the data from the camera to preview buffer (640x480). Then the selected values are copied to the smalled output buffer (see the next paragraph for the method implementation). Output colors are swapped to correct XNA's Texture2D format and finally the output buffer is returned. |
| ? | + | ? |
| ? | + | The output buffer is a linear array of integer values. Every integer has the compressed info about the whole RGB value of the pixel and its alpha (transparency). In the field outputSize we have the X and Y size of the output image. |
| ? | + | ? |
| ? | + | === Helper methods === |
| ? | + | ? |
| ? | + | Now we can design the method for copying values from the preview to the output buffer. Here you can find the most complex version. This method automatically crops the image by the correct output ratio. The image is also auto-rotated and scaled. |
| ? | + | ? |
| ? | + | <code csharp> |
| ? | + | private void CopyValuesToOutputBuffer() |
| ? | + | { |
| ? | + | ? ? // Copies data from preview buffer (640x480) to smaller output buffer in correct ratio |
| ? | + | ? ? Point start = Point.Zero; |
| ? | + | ? ? Point incr = new Point(previewSize.X / outputSize.Y, previewSize.Y / outputSize.X); |
| ? | + | ? |
| ? | + | ? ? if (previewSize.X / (float)previewSize.Y > outputSize.Y / (float)outputSize.X) |
| ? | + | ? ? { |
| ? | + | ? ? ? ? // Preview is wider, output buffer will be cropped at left/right side |
| ? | + | ? ? ? ? start.X = (int)((previewSize.X - outputSize.Y * previewSize.Y / (float)outputSize.X) / 2); |
| ? | + | ? ? ? ? incr.X = (previewSize.X - 2 * start.X) / outputSize.Y; |
| ? | + | ? ? } |
| ? | + | ? ? else |
| ? | + | ? ? { |
| ? | + | ? ? ? ? // Output buffer is wider (preview is taller, crop at top/bottom) |
| ? | + | ? ? ? ? start.Y = (int)((previewSize.Y - outputSize.X * previewSize.X / (float)outputSize.Y) / 2); |
| ? | + | ? ? ? ? incr.Y = (previewSize.Y - 2 * start.Y) / outputSize.X; |
| ? | + | ? ? } |
| ? | + | ? ? ? ? ? ? |
| ? | + | ? ? // Inserts values to the output buffer |
| ? | + | ? ? for (int y = 0; y < outputSize.Y; y++) |
| ? | + | ? ? ? ? for (int x = 0; x < outputSize.X; x++) |
| ? | + | ? ? ? ? { |
| ? | + | ? ? ? ? ? ? // Auto flip / rotate the image to output |
| ? | + | ? ? ? ? ? ? int sourceX = Math.Min(start.X + y * incr.X, previewSize.X - 1); |
| ? | + | ? ? ? ? ? ? int sourceY = Math.Min(previewSize.Y - (start.Y + x * incr.Y) - 1, previewSize.Y - 1); |
| ? | + | ? ? ? ? ? ? int i = sourceX + sourceY * previewSize.X; |
| ? | + | ? ? ? ? ? ? outputBuffer[x + y * outputSize.X] = previewBuffer[i]; |
| ? | + | ? ? ? ? } |
| ? | + | } |
| ? | + | </code> |
| ? | + | ? |
| ? | + | {{Note|We are using the integer division for calculating the increment (for performance reasons). This will work fine if the output buffer is sufficiently smaller than the camera preview. If the output and preview sizes are almost similar, this method may give you poor results.}} |
| ? | + | ? |
| ? | + | ? |
| ? | + | The next method converts the image buffer to the XNA format: |
| ? | + | ? |
| ? | + | <code csharp> |
| ? | + | private void SwapRedBlueChannel(int[] buffer) |
| ? | + | { |
| ? | + | ? ? for (int i = 0; i < buffer.Length; ++i) |
| ? | + | ? ? { |
| ? | + | ? ? ? ? // Converts a packed RGB integer value from Silverlight's to XNA format |
| ? | + | ? ? ? ? // - we need to switch a red and blue channel |
| ? | + | ? ? ? ? buffer[i] = (int)((uint)buffer[i] & 0xFF00FF00) |
| ? | + | ? ? ? ? ? ? ? ? | (buffer[i] >> 16 & 0xFF) |
| ? | + | ? ? ? ? ? ? ? ? | (buffer[i] & 0xFF) << 16; |
| ? | + | ? ? } |
| ? | + | } |
| ? | + | </code> |
| ? | + | ? |
| ? | + | And finally, we should include to our CamReader class the method for stopping the camera. It will be called when we leave the app, or switch it to background: |
| ? | + | ? |
| ? | + | <code csharp> |
| ? | + | public void StopCamera(Canvas fakeCamCanvas) |
| ? | + | { |
| ? | + | ? ? if (camera != null) |
| ? | + | ? ? ? ? camera.Dispose(); |
| ? | + | ? ? camera = null; |
| ? | + | ? ? previewBuffer = null; |
| ? | + | ? ? fakeCamCanvas.Background = null; |
| ? | + | } |
| ? | + | </code> |
| ? | + | ? |
| ? | + | == Integration == |
| ? | + | ? |
| ? | + | xxx yyy zzz |
| ? | | ? | |
| ? | == Summary == | ? | == Summary == |
This article explains how to efficiently read a live stream from the camera on Windows Phone 7 and 8 devices and how to apply an effect filter on this stream.
Most of the photo apps on the market are pretty simple. They allow users to take a photo, then apply some interesting vintage filter and save the photo. But only a few applications have a true live viewfinder with a real-time applied filter. Would it be nice to see how the photo will look like, even before you press the shutter button? What about some other live effects, like a pixelated 8-bit style screen?
We will show you in this article how to make an app (in C# language) that can run on both Windows Phone 7 and Windows Phone 8 devices. Our goal will be to design a sufficiently fast implementation that can run flawlessly even on the oldest and slowest devices. We will go through several ideas and implementations and we will compare them.
We will be demonstrating these methods on the Windows Phone 7 (WP7) project. In the last chapter we will cover the differences between WP7 and Windows Phone 8 (WP8):
The most simple solution for showing a direct stream from the camera is the VideoBrush component. The code is the same on WP7 and WP8. You can add this code into your XAML page:
In the code-behind you can initialize the camera and set the stream to the VideoBrush:
Do not forget to add an ID_CAP_ISV_CAMERA capability. This is a very simple solution. The stream from the camera is quite distorted and wrongly rotated, but you can fix this by setting a RelativeTransform on the Rectangle. But what is a real pity, you can't modify this stream in any way. You can't apply any filter or effect on the VideoBrush object, you don't have an access to the raw data of the pixels.
We will use a different approach. As you can discover, there are some GetPreviewBuffer methods on the PhotoCamera object. This is the way how we can get the raw data from the camera. But we would like also to display this data somewhere, in some fast and efficient way. WritableBitmap component from the Silverlight is very slow, writing data pixel by pixel using this component can take a hundreds of milliseconds. We will need to find another solution.
We will create a new combined Silverlight/XNA project for Windows Phone 7 (we will talk about WP8 later). We would like to combine the ease of XAML and Silverlight with the power of the XNA in one solution. We will use a very similar approach to one described in the article Use Camera with XNA.
You need to have installed a Windows Phone SDK 7.1. Open your Visual Studio 2010 (Express or the Professional/Ultimate) and create a new Windows Phone Silverlight and XNA Application project.
To be able to read from the camera we need to place a fake Canvas to our XAML code. Add to GamePage.xaml page this line of code:
We will create a helper object now. Lets create a new class CamReader.cs and place into it these declarations:
As you can see, there is some preview buffer and the output buffer. The preview buffer will contain the raw values from the camera and will be usually 640x480 pixels big. Applying effects on this large resolution would be very resource intensive and slow. We do not need such a big resolution for the live preview. Our output buffer will be smaller, for example only 240x240 px. All computations and filter calculations will be executed on this smaller resolution. We can scale up this image later, it will be a little blurred, but it does not matter much for us.
This method retrieves the data from the camera to preview buffer (640x480). Then the selected values are copied to the smalled output buffer (see the next paragraph for the method implementation). Output colors are swapped to correct XNA's Texture2D format and finally the output buffer is returned.
The output buffer is a linear array of integer values. Every integer has the compressed info about the whole RGB value of the pixel and its alpha (transparency). In the field outputSize we have the X and Y size of the output image.
Now we can design the method for copying values from the preview to the output buffer. Here you can find the most complex version. This method automatically crops the image by the correct output ratio. The image is also auto-rotated and scaled.
And finally, we should include to our CamReader class the method for stopping the camera. It will be called when we leave the app, or switch it to background:
The "platform categories" will be displayed here in preview only - Copy paste relevant categories into text here
Windows Phone: [[Category:Windows Phone]]
[[Category:Windows Phone 7.5]]
[[Category:Windows Phone 8]]
Nokia Asha: [[Category:Nokia Asha]]
[[Category:Nokia Asha Platform 1.0]]
Series 40: [[Category:Series 40]]
[[Category:Series 40 1st Edition]] [[Category:Series 40 2nd Edition]]
[[Category:Series 40 3rd Edition (initial release)]] [[Category:Series 40 3rd Edition FP1]] [[Category:Series 40 3rd Edition FP2]]
[[Category:Series 40 5th Edition (initial release)]] [[Category:Series 40 5th Edition FP1]]
[[Category:Series 40 6th Edition (initial release)]] [[Category:Series 40 6th Edition FP1]] [[Category:Series 40 Developer Platform 1.0]] [[Category:Series 40 Developer Platform 1.1]] [[Category:Series 40 Developer Platform 2.0]]
Symbian: [[Category:Symbian]]
[[Category:S60 1st Edition]] [[Category:S60 2nd Edition (initial release)]] [[Category:S60 2nd Edition FP1]] [[Category:S60 2nd Edition FP2]] [[Category:S60 2nd Edition FP3]]
[[Category:S60 3rd Edition (initial release)]] [[Category:S60 3rd Edition FP1]] [[Category:S60 3rd Edition FP2]]
[[Category:S60 5th Edition]]
[[Category:Symbian^3]] [[Category:Symbian Anna]] [[Category:Nokia Belle]]
Add categories below using category selector.
Source: http://developer.nokia.com/Community/Wiki/index.php?title=Real-time_live_camera_view_with_effects_for_Windows_Phone_7_and_8&diff=205076&oldid=205036
Stacie Halas Corvette Stingray Claire Danes Amy Poehler Australian Open Girls Hbo Golden Globes
