Presents current trends and potential future developments by leading researchers in immersive media production, delivery, rendering and interaction

The underlying audio and video processing technology that is discussed in the book relates to areas such as 3D object extraction, audio event detection; 3D sound rendering and face detection, gesture analysis and tracking using video and depth information. The book will give an insight into current trends and developments of future media production, delivery and reproduction. Consideration of the complete production, processing and distribution chain will allow for a full picture to be presented to the reader. Production developments covered will include integrated workflows developed by researchers and industry practitioners as well as capture of ultra-high resolution panoramic video and 3D object based audio across a range of programme genres. Distribution developments will include script based format agnostic network delivery to a full range of devices from large scale public panoramic displays with wavefield synthesis and ambisonic audio reproduction to small screen mobile devices. Key developments at the consumer end of the chain apply to both passive and interactive viewing modes and will incorporate user interfaces such as gesture recognition and second screen devices to allow manipulation of the audio visual content.

Oliver Schreer, Fraunhofer Heinrich-Hertz-Institut, Berlin, Germany
Dr. Schreer is Scientific Project Manager of the "Immersive Media& 3D Video" Group in the Image Processing Department at HHI, Berlin. He is responsible for the lead of European Research projects (e.g. FascinatE project) and coordinating research on 3D video processing and immersive media.

Graham Thomas, BBC Research& Development, London, UK
Dr. Thomas is Principal Research Engineer and Section Lead of the Production Magic department at BBC R&D. He leads a team of engineers developing 3D image processing and graphics techniques for TV production. His work has led to many commercial products, e.g. the Piero sports graphics system, which won a Queens Award in 2011. He holds 20 patents and is Visiting Professor at University of Surrey.

Ben Shirley, University of Salford, UK
Ben Shirley is Senior Lecturer at the University of Salford and Academic Lead for the Digital Innovation Cluster in the Universitys College of Science& Technology at MediaCityUK. He has been principal investigator for a range of broadcast related research projects including leading Salfords work in the EU FascinatE project. He teaches audio technology and broadcast engineering at undergraduate and postgraduate level.

Georg Thallinger, Joanneum Research, Graz, Austria
Dr. Thallinger is Head of the research group "Audiovisual Media" at JOANNEUM RESEARCH, Austria. He oversees a group active in the field of digital media with a focus on video and image processing as well as metadata for applications in media production and archiving, film restoration, media monitoring and security. Dr. Thallinger is involved in the EU FascinatE project.

Jean-Francois Macq, Alcatel-Lucent, Antwerp, Belgium
Dr. Macq is a Senior Research Engineer at Bell Labs, the research organization of Alcatel-Lucent. He has been responsible for several national and international research projects in the field of video coding and delivery, currently including the EU FascinatE project.

Javier Ruiz-Hidalgo, Universitat Politècnica de Catalunya (UPC), Barcelona, Spain
Dr. Ruiz-Hidalgo is an Associate Professor at the Universitat Politècnica de Catalunya. He has been involved in various European Projects as a principal researcher from the Image Processing Group at UPC, including the EU FascinatE project.

Omar Niamut, Netherlands Organization for Applied Scientific Research (TNO), Delft, The Netherlands
Dr. Niamut is a Senior Research Scientist in the "Media Networks and Services" Group at TNO. He has advised the European Parliament on the harmonization of mobile TV and was editor of the technology report on mobile TV in the European Commission-assigned European Mobile Broadcast Council. He participated in ETSI TISPAN standardization with over 200 contributions on interactive services for next generation IPTV and advised the Singapore government on the use of IPTV standards. He has been involved in a number of European projects, including the Fascinate project. Dr. Niamut holds 15 patents.

List of Editors and Contributors xiii

List of Abbreviations xvii

Notations xxiii

1 Introduction 1
Oliver Schreer, Jean-Francois Macq, Omar Aziz Niamut, Javier Ruiz-Hidalgo, Ben Shirley, Georg Thallinger and Graham Thomas

2 State-of-the-Art and Challenges in Media Production, Broadcast and Delivery 5
Graham Thomas, Arvid Engstrom, Jean-Francois Macq, Omar Aziz Niamut, Ben Shirley and Richard Salmon

2.1 Introduction 5

2.2 Video Fundamentals and Acquisition Technology 7

2.2.1 How Real Should Video Look? 7

2.2.2 Fundamentals of Video 9

2.2.3 Camera Technology 14

2.2.4 Production for Formats with Differing Aspect Ratios 19

2.2.5 Stereoscopic 3D Video 20

2.2.6 Challenges for the Future 21

2.3 Audio Fundamentals and Acquisition Technology 21

2.3.1 Introduction 21

2.3.2 Fundamentals of Audio 21

2.3.3 Non-Live Production 27

2.3.4 Live Production 31

2.3.5 Surround Sound 34

2.3.6 Challenges for the Future 34

2.4 Live Programme Production 34

2.4.1 The Production Area and Roles in Production 35

2.4.2 The Production Workspace 37

2.4.3 Vision Mixing: Techniques for Directing, Selecting and Mixing Camera Feeds 38

2.4.4 Audio Mixing 40

2.4.5 Replay Operation in Live Television Production 40

2.4.6 Challenges for the Future 42

2.5 Coding and Delivery 43

2.5.1 Managed Delivery Networks 43

2.5.2 Unmanaged Delivery Networks 47

2.5.3 Codecs and Transport Protocols 48

2.5.4 Challenges for the Future 50

2.6 Display Technology 50

2.6.1 Plasma Displays Developing the Flat Panel Display Market 51

2.6.2 LCD the Current Dominant Technology 52

2.6.3 Projection Technologies 53

2.6.4 Other Technologies 54

2.6.5 Impact on Broadcasters 55

2.6.6 Challenges for the Future 56

2.7 Audio Reproduction Technology 56

2.7.1 Stereophonic Sound Systems 57

2.7.2 Holophonic Systems 59

2.7.3 Binaural Systems 61

2.7.4 Hybrid Systems 62

2.7.5 Challenges for the Future 62

2.8 Use of Archive Material 62

2.8.1 Video Format Conversion 63

2.8.2 Audio Format Conversion 64

2.8.3 Challenges for the Future 64

2.9 Concept of Format-Agnostic Media 65

2.9.1 Limitations of Current Production and Delivery Approaches 65

2.9.2 A New Approach: Format-Agnostic Media 65

2.9.3 Metadata for Enabling Rich User Interaction 66

2.9.4 A Format-Agnostic Media Production and Delivery System 67

2.10 Conclusion 68

Notes 69

References 69

3 Video Acquisition 74
Oliver Schreer, Ingo Feldmann, Richard Salmon, Johannes Steurer and Graham Thomas

3.1 Introduction 74

3.2 Ultra-High Definition Panoramic Video Acquisition 75

3.2.1 History of Panoramic Imaging 75

3.2.2 The Geometry of Two Views 79

3.2.3 Fundamentals of Panoramic Video Acquisition 82

3.2.4 Geometrical Constraints for Parallax Free Stitching of Two Images 84

3.2.5 Registration of Views 88

3.2.6 Stitching, Warping and Blending of Views 91

3.3 Use of Conventional Video Content to Enhance Panoramic Video 94

3.3.1 Calibration of Camera Position and Orientation 94

3.3.2 Photometric Matching of Panoramic and Broadcast Cameras 98

3.3.3 Blending and Switching Between Camera Views 101

3.4 High Frame Rate Video 102

3.4.1 Early Work on HDTV Frame Rates 104

3.4.2 Issues with Conventional Frame Rates 104

3.4.3 Practical Investigations into the Effects of High Frame Rates 107

3.4.4 Future Frame Rates for TV Production and Distribution 111

3.4.5 Consideration of Frame Rates and Motion Portrayal in Synthetic Production 111

3.4.6 Conclusions on Frame Rates 112

3.5 High Dynamic Range Video 112

3.5.1 The Human Visual System in Natural Environments 113

3.5.2 Conventional and HDR Video Cameras 115

3.5.3 Conventional and HDR Displays 117

3.5.4 HDR Video Formats 119

3.5.5 Adaptive Tone-mapping for Format-Agnostic Video 120

3.6 Conclusion 125

Notes 126

References 126

4 Platform Independent Audio 130
Ben Shirley, Rob Oldfield, Frank Melchior and Johann-Markus Batke

4.1 Introduction 130

4.2 Terms and Definitions 132

4.2.1 Auditory Event and Sound Event 132

4.2.2 Basic Room Acoustics Theory 134

4.3 Definition of the Problem Space 135

4.3.1 Reproduction Environment 135

4.3.2 Reproduction Method 137

4.3.3 Audio-visual Coherence 141

4.3.4 User Interaction 143

4.3.5 Example Scenario 143

4.4 Scene Representation 144

4.4.1 Components of a Virtual Sound Scene 144

4.4.2 Representations of Virtual Sound Scenes 146

4.4.3 Implementation Examples 147

4.5 Scene Acquisition 149

4.5.1 Capturing Discrete Audio Objects 150

4.5.2 Capturing the Sound Field Component 152

4.5.3 Capturing the Diffuse Field 153

4.6 Scene Reproduction 153

4.6.1 Scenario: Mobile Consumption Via Headphones 153

4.6.2 Scenario: Interactive Multichannel Reproduction 154

4.6.3 Scenario: Big Screen 154

4.6.4 Scenario: Interactive and Free Viewpoint 3D 155

4.7 Existing Systems 156

4.7.1 Commercial Systems 156

4.7.2 Research Projects 157

4.7.3 Perceptive Media 160

4.8 Conclusion 161

4.8.1 Open Issues 162

References 162

5 Semi-Automatic Content Annotation 166
Werner Bailer, Marco Masetti, Goranka Zoric, Marcus Thaler and Georg Thallinger

5.1 Introduction 166

5.1.1 Requirements on Semi-automatic Annotation Tools 167

5.1.2 Requirements on Metadata 168

5.2 Metadata Models and Analysis Architectures 170

5.2.1 Metadata Models 170

5.2.2 Architectures for Audio-visual Analysis 171

5.2.3 Storing MPEG-7 Metadata 172

5.2.4 Bulk Loading Techniques for Massive MPEG-7 Metadata Storage 175

5.2.5 An Example Architecture of a Semantic Layer Management System 175

5.3 Domain-independent Saliency 177

5.3.1 Spatio-temporal Visual Saliency 177

5.3.2 Estimating Grid-based Saliency 178

5.3.3 Salient Regions for Controlling Automated Shot Selection 179

5.4 Person Detection and Tracking 180

5.4.1 Person Detection 181

5.4.2 Person Tracking 182

5.4.3 Multicamera and Panoramic Environment 184

5.4.4 GPU Accelerated Real-time Tracking Beyond HD 187

5.5 Online Detection of Concepts and Actions 189

5.5.1 Sequence-based Kernels 190

5.5.2 Kernels for Online Detection 193

5.5.3 Performance of Online Kernels 194

5.6 Supporting Annotation for Automated Production 195

5.6.1 User Preferences and Functionality Definition 195

5.6.2 Design Overview and Principles 196

5.6.3 Preconfiguration and Predefined Workspaces 198

5.7 Conclusion 204

References 205

6 Virtual Director 209
Rene Kaiser and Wolfgang Weiss

6.1 Introduction 209

6.1.1 What is a Virtual Director? 210

6.1.2 Features Enabled by a Virtual Director 211

6.1.3 Definitions 212

6.1.4 Requirements for Virtual Director Technology 213

6.1.5 Existing Implementations and Research Activities 215

6.2 Implementation Approaches 219

6.2.1 Behaviour Implementation Approaches 220

6.2.2 Combining Rule Engines and Event Processing Technology 223

6.3 Example Architecture and Workflow 225

6.3.1 Workflow of the Production System 225

6.3.2 Workflow of the Virtual Director 226

6.3.3 Distributed Nature 228

6.3.4 Sources of Knowledge 228

6.4 Virtual Director Subprocesses 230

6.4.1 Semantic Lifting 230

6.4.2 Shot Candidate Identification 231

6.4.3 Shot Framing 233

6.4.4 Shot Prioritisation 234

6.4.5 Decision Making 235

6.5 Behaviour Engineering: Production Grammar 237

6.5.1 Production Knowledge Elicitation Process 237

6.5.2 Cinematic Techniques 238

6.5.3 Audio Scripting 241

6.5.4 Domain Model 241

6.5.5 Limitations in Rule-based Behaviour Engineering 242

6.6 Virtual Director: Example Prototype 243

6.6.1 Architecture and Software Framework 245

6.6.2 Production Scripts 246

6.6.3 Behaviour Implementation 247

6.6.4 Production Grammar Example 248

6.7 Conclusion 251

6.7.1 Summary 251

6.7.2 Limitations 251

6.7.3 Testing and Evaluation 252

6.7.4 Research Roadmap 253

6.7.5 Concluding Thoughts 255

References 256

7 Scalable Delivery of Navigable and Ultra-High Resolution Video 260
Jean-Francois Macq, Patrice Rondao Alface, Ray van Brandenburg, Omar Aziz Niamut, Martin Prins and Nico Verzijp

7.1 Introduction 260

7.2 Delivery of Format-Agnostic Content: Key Concepts and State-of-the-Art 262

7.2.1 Delivery Agnostic to Content Formats, Device Capabilities and Network Bandwidth 262

7.2.2 Delivery Agnostic to Video Timing the Temporal Interactivity Case 264

7.2.3 Delivery Agnostic to Video Reframing the Spatial Interactivity Case 266

7.3 Spatial Random Access in Video Coding 267

7.3.1 Video Compression and Random Access A Fundamental Trade-off 268

7.3.2 Spatial Random Access by Tracking Coding Dependencies 271

7.3.3 Spatial Random Access by Frame Tiling 271

7.3.4 Multi-Resolution Tiling 273

7.3.5 Overlapping Tiling for Low-Powered Devices 274

7.4 Models for Adaptive Tile-based Representation and Delivery 276

7.4.1 Saliency-based Adaptive Coding of Tiled Content 277

7.4.2 Optimisation of Tile Selection Under Delay and Bandwidth Constraints 280

7.5 Segment-based Adaptive Transport 281

7.5.1 Video Streaming Over IP 282

7.5.2 Tiled HTTP Adaptive Streaming Over the Internet 285

7.5.3 Publish/Subscribe System for Interactive Video Streaming 289

7.6 Conclusion 294

References 294

8 Interactive Rendering 298
Javier Ruiz-Hidalgo, Malte Borsum, Axel Kochale and Goranka Zoric

8.1 Introduction 298

8.2 Format-Agnostic Rendering 299

8.2.1 Available Rendering Solutions in End Terminals 299

8.2.2 Requirements for Format-Agnostic Video Rendering 306

8.2.3 Description of a Technical Solution 308

8.3 Device-less Interaction for Rendering Control 311

8.3.1 Sensors for Gesture Recognition 314

8.3.2 Gesture Analysis Techniques 317

8.3.3 Recognition and Classification Techniques 319

8.3.4 Remaining Challenges in Gesture Recognition Systems 321

8.3.5 Description of a Technical Solution 321

8.3.6 User Study of the Gesture Interface 327

8.4 Conclusions 331

References 332

9 Application Scenarios and Deployment Domains 337
Omar Aziz Niamut, Arvid Engstrom, Axel Kochale, Jean-Francois Macq, Graham Thomas and Goranka Zoric

9.1 Introduction 337

9.2 Application Scenarios 338

9.2.1 Digital Cinema: A Totally Immersive Experience 338

9.2.2 Home Theatre: In Control 339

9.2.3 Mobile: Navigation and Magnification 339

9.3 Deployment in the Production Domain 340

9.3.1 Outlook on New Production Practices 340

9.3.2 Use of Format-Agnostic Technology to Aid Conventional Production 342

9.3.3 Production of Format-Agnostic Content to Support End User Interaction 344

9.4 Deployment in the Network Domain 347

9.4.1 Network Requirements 347

9.4.2 Impact of Application Scenarios 348

9.5 Deployment in the Device Domain 351

9.5.1 Device Capabilities 351

9.5.2 User and Producer Expectations 355

9.6 Deployment in the User Domain 356

9.7 Conclusion 357

References 357

Index 359