Archive for May, 2011

Why the LinkedIN is so great!

Tuesday, May 31st, 2011

Happy New Year to everyone. Solde adidas superstar I just want to share a joyous event with you, Chaussure Adidas Pas Cher I won’t comment it at all, Fjallraven Kanken Pas Cher but just hang the pictures there for you…

 

LinkedIN post on PlayKontrol

Reaction to the post

[caption id="attachment_419" align="aligncenter" width="337" caption="OK,

Treatment of repeating content

Saturday, May 28th, 2011

In media monitoring systems and environments, we often have to identity and COUNT the occurrences of some playback event. Most common examples of such are when you have to monitor all the occurrences of the same commercial audio spot.

Multiple parties are interested in tracking audio spots:

  • broadcasters
  • advertisers (clients)
  • agencies (clients representative)
  • government regulators

Let me briefly cover what do they need to know about playback of the commercial audio spots.

Broadcasters

They need proof that they played something at a certain time, to show it to the client and be able to issue invoices for services provided.

Advertisers and agencies

They both need to have a proof that something was played – their own commercials, at certain times, and by correct amount. However, they might also need to be able to look into other brands so they can track their competitors.

Government regulators

They usually want to know if the proposals or laws requirements on the broadcast media is met. New Balance 574 Pas Cher Such requirements are for example to have no more then 2 minutes of advertisements per hour, or to have commercial blocks clearly separated from the rest of the program by special markers called ‘jingles’ or ‘breaks’.

Let’s get back to..

The problem

Usual workflow for the above is to fill the matching technology with a samples that you want to track, and the technology will give you the locations in the timeline for the samples provided. Adidas Femme That is one thing that PlayKontrol can do for you. sac fjallraven kanken But, what if you don’t have the samples, and still want to discover them?

The rescue

Traditional way would be to go through the known parts of the program, mark them, clip them out and have audio spotter search for all of the occurrences. With that method, and with lot of clipping, you’ll have some accuracy, and some clips will miss you attention because they aren’t in their place, for example commercial is out of its commercial block.

Other way is to do it with PlayKontrol SelfMatching technology. Adidas ZX Flux Heren It works in a way that whole day of archive is given to the PK, and the result of the process is a list that contains ALL of the matches for the given day.

So every repeating audio clip, no mater how small, will be listed here. Nike Air Max 2016 Heren grijs From there, your analyst only task would be to:

  • browse through the clips,
  • listen to them,
  • maybe fine-clip them,
  • tag them and
  • put them into the repository.

I have created a picture containing the results of the process in ‘visual representation’. nike air max pas cher Here it is:

Please note the following:

  • both X and Y axes represent time
  • grid divides time in one-hour interval
  • grayed areas are time intervals 00-06 and 18-24 (say night time)
  • size of the points represent length of the clip that is matched

Try to figure out the rest for yourself.

Capturing and archiving of DVB-T signal

Saturday, May 28th, 2011

No matter if it’s for compliance recording so you will capture and save your own broadcast, or you are doing media monitoring and you would like to capture multiple signals of the air, you have some interesting choices here.

Let’s explore in detail your options on the subject, whether it’s one channel or multiple channel recording.

One channel DVB-T recorder

Recording of one channel is simple no matter how you choose to record it. Let me present two main options here for you, so you could see what is most applicable in your situation.

Simplest way of recording would be to have one set top box for DVB-T, and use it to send composite signal into the computer via the Osprey 210 card. It is the most robust solution, but it has some (serious) drawbacks:

  • cheap DVB-T tuners can ‘lock’ and freeze the picture
  • low quality tuners can also de-sync audio and video with time – and you need 24/7 operation here
  • you’ll need extra power connector for the set top box
  • STB-s are producing extra heat

Alternative way of recording is to use DVB-T card such as Asus MyCinema-ES3-110, use software such as TubeSink to tune on a frequency and extract the channel required from it (this is called DEMUX-ing) and forward the extracted channel to the VideoPhill Recorder for further processing (recording, streaming, …).

BTW, TubeSink mentioned above can be used even without VideoPhill Recorder, as it DEMUXes the channels and can forward them to any computer on your network as an UDP Transport Stream that can be playable with VLC. Nike Air Max 2017 Heren grijs It you want to use it for non-commercial purposes, download it from here.

So in the case on 1 channel DVB-T recording, I would say that it remains uncertain whether to use external set top box with Osprey capture card, or go with pure software solution and some simple of-the-shelf DVB-T tuner.

But in case of…

Multiple channels DVB-T recording facility

Same options are available at multiple channel recording facilities – but here is the catch. As you might probably know, multiple DVB-T channels are packed and are transmitted at one frequency and that is called multiplexing. The carrier for the channels that are transmitted is called MULTIPLEX (MUX for short). nike air max 2017 pas cher In several occasions it has 4 channels, and sometimes it can have as much as 16 or more channels.

Current recommended recorder density (channels per machine) is 4. Adidas Gazelle Heren One machine packed with Osprey 460e will do 4 channels just fine.

So, let’s say that we need 16 channels and they are scattered across 3 MUX-es (we have such situation here in Zagreb). Using a conventional method (I would say that having 16 STBs is conventional, as bizarre as it seems) you’ll need the:

  • 4 recording servers
  • 4 Osprey 460e cards
  • 16 DVB-T set top boxes
  • PLENTY of mains outlets
  • some kind of distribution to have the signal distributed to all 16STBs

Since you see where I’m coming to, let me suggest the following; let’s use TubeSink to control 3 tuners in TWO MACHINEs, and save on 4 Ospreys and 2 PCs, and the rest of the unnecessary equipment.

We’ll put 2 tuners into one machine, and one tuner in the second machine. kanken fjallraven If the channel per MUX distribution is such that each machine has it’s 8 channels, fine. Nike Air Huarache Heren If not, we’ll instruct TubeSink to forward the Transport Stream to ANOTHER machine and that machine will perform recording. In that way, load will be completely balanced between two machines, and you’ll have your 16 channels recorder in a nice and compact fashion.

Even more compact?

Yes, it can go even further. There are dual DVB-T tuners such as WinTV-HVR-2200 that can provide tuning to two frequencies at once, and with it, you could record as much channels there are in two MUXes at one machine. Nike Air Max 2017 Homme Today, even desktop processors such as i7 can encode 8 channels of video in real time. So, with proper CPU (or multiple CPUs on server computers) – even 16 channels could be encoded in one compact 2U rack mounted unit.

However… (serious problem)

Using PC based DVB-T cards will only work with free to air channels.

SD-SDI compliance recording

Saturday, May 28th, 2011

I will try to write something about creating the archive for the compliance recording purposes from the SD-SDI source. This post is in a response from an repeated inquiries about system that would create such an archive.

Compliance recording in general

Just to remind us – compliance recording serves only one purpose – to prove or disprove that something went on the air at some time. That is the first and the last thing that this technology is used for. nike air max 2017 goedkoop There is no any requirement on the content of the signal – it just have to be good enough for someone to see basic stuff that’s in there .

This one business requirement is countered from the other side with need to create the archiving system to be as affordable as possible. And the cost of the system, if we count out the software involved (system and application part) is highly dependent on the:

  • picture quality
  • number of channels recorded
  • number of possible outputs needed
  • and days of the archive that should be kept

Picture (ie. video) quality is directly proportional to the BITRATE of the video that is recorded. More bitrate, more apparent quality, both in picture resolution, object motion, and so on. cheap fjallraven kanken uk For example, most recorded videos that you might have on your computer are in between 700 and 1000 kbit and are recorded with DivX or similar encoder. Watching a movie at 1000 kbit bitrate is highly enjoyable, and everything above that is required ONLY for HD or HD-Ready content.

Compliance recording means recording what’s on the air

Many people forget that when you do compliance recording, you have to record what was coming out of your transmitter array, not the final that you are broadcasting. In case your link went down, or your output HF amplifier burned, your endpoint picture will be NOTHING, and if you are recording your output, there will be a great discrepancy in your logs.

So suggestion: drop the thinking about recording SD-SDI, and try to record what comes back from the air, in the form of the analog or DVB-T signal. That is the REAL broadcast that your viewers see.

The conflict of interest

We all want stuff to be as cheap as possible. In order to build a recording system that is affordable, we have to balance between various things, but here I’ll try to explore the differences when SD-SDI signal capture is required.

I will postulate this input parameters for the archive that will serve as example here:

  • video bitrate: 512 kbit
  • audio bitrate: 64 kbit
  • archive duration (number of days to keep from today): 92 (3 months)
  • number of channels that needs recording: 4
  • required outputs: both WMV (Windows Media Video) and h.264 (at the same time)
  • recording resolution: 3/4 of PAL D1 picture size so: 540×432

In order to calculate hard drive space requirements for this here, I’ll use the online bitrate calculator here on this blog.

Table below taken from the calculator itself.

Video Bitrate Audio Bitrate Total Bitrate Days Channels Disk Space
512 kbit/s 64 kbit/s 2304 kbit/s 92 4 2183.20 GB

So we need 2200GB of net drive space. In order to provide that kind of space with some fault tolerance, I would recommend having two drives such as WD CAVIAR GREEN 2500GB connected in a MIRROR VOLUME (RAID1). There is no need for additional system drive, and almost all new motherboards can provide on-board implementation of RAID1 – mirroring.

I’m kind of beating around the bush here – because main point of this article should be to provide you with a choice of whether to use original SD-SDI signal or convert to composite signal (or use the signal that comes back).

Osprey vs DeckLink

The choice that lies before us is either to use Osprey 460e card, or a DeckLink Quad card. List price for the Osprey is about $1200, and two DeckLink Quad card is about $1000. So, the first impression is that you’ll go cheaper with DeckLink. But…

So far, the Osprey has proven itself to be absolute master of video capture. Nike France Every system deployed so far didn’t have any problems with the card whatsoever. Tennis Nike France You plug it in, install drivers, and it works. adidas shoes uk And with Osprey 460e, you’ll use only ONE PCI-e slot. Fjallraven Kanken Big Other slot, if there is any, will be used by the graphics card.

Same thing about the slots applies to DeckLink as well. However, there is “now shipping” image below the DeckLink Quad card name, and it implies something – it is red-hot new.

Recording multiple FM radio stations (works for AM, too)

Friday, May 27th, 2011

As it seems, we in media monitoring want to record everything. nike air max pas cher Good part of everything is still in FM radio spectrum (or AM in some flat-land countries). An usually, there are plenty of stations on the air that we have to record, at least a dozen at a given location…

Ancient history

Many many years ago, when I was working in FirePlay (great radio automation company and software) we had a task to produce a recorder that would record ONE channel of radio program 24/7. At that time, encoding MP3 in real-time was some kind of science, and wasn’t available but on most advanced systems that were available (I won’t try to be exact here, but it was something on the lines of Pentium 133Mhz).

So we build FireSave, first version, that was able to handle 1 channel and record it to hard drive, encoded in mp3 format. We even tried to use some obscure GSM codecs to save space even more…

Ancient history, but without dinosaurs

Setup above required live external tuner to be connected to the Sound Blaster (yeah, really). We had some multi-channel cards but they were expensive, and using them to record a confidence and/or compliance recording would be waste of money.

Our need was expanded from one channel to several, say 4. Since we had some expertise running multiple channels, we quickly added more external tuners, replaced Sound Blaster with some multi-channel monster (it was Wave4, then Gina24, then other stuff from EchoAudio, such as Layla 3G) and finally upgraded the software so it could handle multiple channels.

It worked, with 4 external tuners attached to one PC, sometimes more, it looked like an octopus.

Present days (year 2009)

OK, but what if you need and want to record 150 radio station that typical country like Croatia has? You’ll be able to get some audio cards that will have up to 16 audio inputs (even mono sound will be OK), but to have that kind of external tuners, that is and could provide some kind of a problem. Scarpe Adidas Italia And yet still, they can’t all be heard in one place, so you’ll have to have multiple recording sites in order to capture everything you need.

Or not?

The simple fact is that every good radio station will have its internet stream so it will be heard on the internet. nike air max 2016 goedkoop And there is a way to capture that stream of the internet and save it to hard drive as you would record it. There are multiple tools on the internet that would allow you to capture internet audio streams, and you just have to choose one of them, and you’ll be able to record any radio that has its stream. Before we created StreamSink, I was extensively using StationRipper for my own purposes, and that was the inspiration that was needed to create very similar tool. It is similar in the respect that it records internet audio (and video) streams, but one thing is very different: all ‘rippers’ including StationRipper are designed to try to cut audio stream at song boundaries, creating a library of songs for the user. On the other side, our task was to create system to record internet streams in multiple formats in the archive format usable by VideoPhill Player.

It isn’t anything special – just a bunch of files named in some fashion and cut at every five minutes, with special care not to lose single byte of a stream while cutting it.

So with that system, recording 100 radio stations on a single computer is as simple as having an good internet connection present. Of course, every stream will be recorded as reliably as the server and the internet permits, and there is nothing you can do about it. When using that method, you must allow yourself to lose some of the archive sometimes, for the unforeseen facts. Again – better radio stations (the stations that you will need 100% of the archive) will have better sources, better distribution servers, and thus your archive will be better covered.

Expected Archive Coverage

But what to do when there are NO streams?

Lately (Summer 2011), there was a client that needed to record multiple radio stations as well. However, after initial investigation we concluded that radio stations that needed recording were either badly presented on the internet or not presented at all. So instead of capturing streams, we were aiming to capture radio signal from the FM directly. All we had was the antenna that was dipped in the airwaves that contained our radio stations (8 of them).

Strategy was as follows: I have a tool that can capture streams in the format that my application (the Player) needs, but we haven’t the streams. Let’s create them.

Shoutcast internet radio is on the market for decades. Goedkoop Adidas Schoenen And it has both free and tremendous support, and their software for creating and distributing internet radio streams are as robust as they can be, since they are field tested in possibly millions of usage scenarios.

As I knew how to encode the stream, how to distribute it (locally) for the StreamSink, I just needed to capture FM signal somehow. Adidas NMD Heren Using 8 external tuners would be funny for the client, and I’ll probably lose them, so I did a little digging and found a beauty in form of a PCI card:

Professional PCI tuner adapter

This little monster (AudioScience ASI8921) is able to capture 8 FM radio channels and give them to the rest of the system in the form of the DirectShow or waveIn API, just what we needed.

Refactoring, it’s fun – part 2

Sunday, May 22nd, 2011

In the first part I tried to set the scene and give you some background of my problem. I’ll try to continue now, and create an interesting and at the same time informative story about performing the refactoring in question.

Most curious of all is that you don’t have to prepare for refactoring. Kanken 20L It can happen no matter what, and the price usually isn’t so high after all. nike air max pas cher Let me remind you – main tool of component design that is dependency injection wasn’t used. Yes, classes do have their responsibilities cleverly defined, and it helps a lot here, because if it weren’t so – whole deal would have to start few steps ‘before’.

I’m not an experienced writer, so I don’t know if I will get the point across, but to me refactoring this was like building level of level of scaffolding, just to use one level to test the next, and at the same time creating scaffolding so it would be used later in a production environment! I guess that there is a name for it, it has to be :)

Step 1:

Creating a duplicate of the main working class, and see how to force the rest of the application to use it when needed.

Say that class name is PCMHasher, and it has following structure (method declarations only):

class PCM_Hash {  public bool Initialize(PCMBufferedReader sourceDataReader);  public uint GetNextHash(); }

My goal was to create alternative class to this. nike air max 1 homme I needed the old class to be able to have some reference to get the results from.

So I created class PCMHash_2. Nike Air Max 2016 Italia That was my first decision – to create same class as before and try to get same results from it, replacing its guts one step at the time.

Using replaced version wasn’t easy, so I took an interface out and derived from it, and created something like:

IPCM_Hash rph; if (!_refactoring)  rph = new PCMHash(); else  rph = new PCMHash_R2();

At this time I would like to re-state the fact that I am in the production all the time, and have to decide on my feet, having to weight out all the implications that would arise from it. nike air max 90 pas cher I am telling that because everyone could see that some dependency injection was to be used here. nike air max 2017 heren grijs But, apart from having to spend much time installing it through the code, I’m not sure how and if it would work anyway.

Why: at a testing time, I want both classes to be able to function side by side.

Refactoring, it’s fun – part 1

Sunday, May 22nd, 2011

It’s a story of refactoring when code that should be refactored isn’t prepared for it a single bit. If I say prepared I guess that it would mean to have test cases, dependency injection code, and so on. Nike Air Max 2017 Dames roze However, I have none of the above in the original code, just the code that works.

Let me explain what I have, what it does, and where it should end. chaussures de foot puma The purpose of this here refactoring session isn’t about having better performance and keep the same functionality – it is to have same algorithm, already proven in various tests work on different data.

Before (a.k.a. nike air max 2017 zwart now):

Component creates PK_HASH from a sound file. By PK_HASH I would mean “code name for our latest tech that can crunch whole audio file to few bytes later to compare that bytes to bytes crunched from the other file and tell you whether it’s the same sound file”. PK stands for PlayKontrol – the brand name.

So, there are few steps to produce the PK_HASH from sound file:

  • decode and read the file – input is file on the disk, for example .mp3, .wma, .aac, and the output are PCM samples
  • from any kind of PCM samples (stereo, mono, 8-bit, 16-bit) we produce array of shorts that must be rewindable
  • hashing the data and producing the PK_HASH files

Decode and read the file

From the file on disk, that can be any file format that is streamable (we produce files with StreamSink – see the archive example here: http://access.streamsink.com/archive/). That is .mp3, .aac, .wma, .ogg, and whatnot.

Currently it’s done by using simple component that uses DirectShow to create a graph for the audio file, renders that graph and attaches SampleGrabber filter to fetch the samples. Component goes from file on the disk to whole PCM sample data into memory. It’s feasible for 5 minute file (5 x 60 x 4 x 44100 = 50MB). It can work even for 1h files. However, you can *feel* that this approach IS wrong, especially when told that for the rest of the algorithm, you don’t have to have access to the whole PCM data at once.

Rewindable sample array

PCM Samples are promoted to 16 bit (if needed), and channels are downmixed to mono. Again, that is done in memory, so for each PCM sample there are 2 bytes of data that are present in memory as a result of the operation.

Hashing and creating the file

Hashing process needs moving and overlapping window over sample array, and since we have everything in the memory, that is a piece of cake now. We take the data, process it, write it into the another byte array. Nike Air Max Goedkoop Since it’s extremely dense now, I won’t cry about memory at this point, but yeah, it is written into the memory first, and then saved to file on disk.

So here I tried to explain how it works so far. It goes from the encoded audio file to PCM sample data in memory, downmixes that data in memory to one PCM channel, processes the mono PCM samples to obtain PK_HASH and then write it to file.

So what do we actually need?

If you take a peek at the archive you’ll find that every folder has audio files, and also has .hash file for ever audio file that is present in the directory. Please note that not every directory is processed, only 20 of those, because processing consumes CPU intensely, and I have only few PCs laying around to scrub the data. Fjallraven Kanken No.2

Will improve in the future. So, for crunching the archive, even POC (proof-of-concept) is OK, as it serve its needs. It will go through the archive and leave processes PK_HASHes.

Process that goes in parallel is waiting for the PK_HASH file to be created, reads it, and does matching against the database. However, next step should be taken, and it is REALTIME processing.

To be able to process in REALTIME, architecture goes somehow like this:

  • StreamSink is attached to the network stream of any kind, and provides PCM sample output
  • PCM sample output is downsampled and buffered
  • hashing process uses buffered mono PCM samples and outputs results into the stream
  • PK_HASH stream is again buffered and results processed with MATCHER process

StreamSink PCM decoding

StreamSink is the application that does internet media stream capture. It can, however, thanks to feature request from DigitalSyphon, process every media stream and provide PCM samples for it in real-time, in a form of the Stream derived class. So, what part of the process is covered completely.

Buffering PCM samples

Now, new component should be created – something that can buffer PCM samples from the Stream and provide floating, overlapping window reads for hashing process. With some thinking I combined inner workings of Circular Buffer stereotype with something that can be used almost directly in the hasher process – by replacing class implementation only.

Processing and creating PK_HASHes

Hasher process was reading the buffered MEMORY quasi-stream. However, it used kind-of simple interface to read the data, so my luck was that that interface could be extracted and implementation done with buffered stream data. Also, output of the class should be rewritten, since it now doesn’t have any Interface-able part to replace.

And so on – later should be implemented from scratch, so there is no story about refactoring here.

Refactoring pyramid/tower

I can call it pyramid or tower, because after long time of procrastination (subconsciously processing the task at hand) I was able to put my hands on the keyboard and start. My premise was that everything has to be checked from the ground up, because NOW I have the algorithm that produces desired results, and since there are many steps involved, an error in a single step could be untraceable if I don’t check every step along the way.

Tools used

I am kind of old fashioned, so this paragraph won’t be very long.

How to reduce hard drive fragmentation

Tuesday, May 17th, 2011

The topic of drive fragmentation might be a little out in this days, but since I spent great deal of my youth watching PC Tools defragment my drive in a graphically pleasing fashion, I am inclined to think that drive fragmentation (when excessive) can severely reduce both computer performance and hard drive life.

As this might be true for the common day-to-day user, it is particularly true for corporate/enterprises that do need their data to be:

  • accessible,
  • quickly accessible,
  • accessible for a long time

In a common computer use scenario, most of the files are there for computer to read an use, either as software that has to be loaded into memory, or documents that have to be shown to the user. Writing to the hard drive is uncommon operation (when you put it against the number of reads) and thus the drive fragmentation however present is in fact easily ignored.

Continuous stream recording, enter…

In my business (my clients businesses’ to be exact) the hard drives are working in opposite. They WRITE all the time, and read only on occasions. And the problem that will surely lead to fragmentation is that in most situations they need to write MULTIPLE long files continuously. adidas 2017 pas cher Let me try to explain what, first from the aspect of why, then move to what…

When either running VideoPhill Recorder for recording video, or using StreamSink to record internet media streams, in most cases user has MULTIPLE channels recorded on one computer. Files that are created by that recording are commonly created at one time (all of them) and are grown continuously until closed. Since Windows is, as it is now, an operating system that can’t reserve drive space in advance (maybe it can, but software doesn’t know how long the files would be) the space for them will be allocated as the time goes by. If we have 4 files that are written slowly but concurrently (and are grown at the same time), we’ll certainly have the following situation on the hard drive (I’m talking ONLY about the data that is stored here, and am simplifying physical hard drive storage as a continuous slate):

file1_block1
file2_block1
file3_block1
file4_block1
file1_block2
file2_block2
file3_block2
file4_block2
.
.
.
file1_blockN
file2_blockN
file3_blockN
file4_blockN

That means fragmentation. fjallraven kanken soldes File isn’t in continuous blocks, but is scattered in evenly and can’t be read sequentially from the hard drive. fjallraven kanken rugzakken You might be lucky and your blocks could be scattered in a way that sectors on the drive will be adjacent and this won’t pose a problem, but what are the chances? :)

And when file1 gets deleted, what remains on the hard drive? A blocks filled with nothing, left there for other files to fill them. New files will try to fill them, and the drive will soon be completely jumbled. It will all be hidden from you by the OS, but still, OS will have to deal with it.

And that is the story of 4 channels. What about situation when you have 60 channels recorded on one machine (I’m talking about internet stream recording, of course). Such an archive could be found here: http://access.streamsink.com/archive/

If you aren’t convinced that this really IS a problem, you can stop reading now.

Rescue #1 – Drive Partitioning

It is feasible in situations where there is low number of channels that needs to be recorded. If you have 4 channels, you’ll create 4 partitions, and each partition will have nice continuous files written to it. Done.

However, you can’t have 50 partitions on one drive and get away with it.

Rescue #2 – Queued File Moving

Other solution for large number of channels presents itself in a form of a temporary partition for initial file recording, and then moving out the files to their permanent location later, but ONE FILE at a time, in a queue.

Queued Moving of Files in StreamSink

This is implemented in StreamSink, and it even has an ability to throttle data rate when moving the files to another drive. adidas schoenen Only thing that is of a problem here is wasting of a temporary hard drive, because it gets beaten by fragmentation.

Rescue #3 – Using RAM Drive on Method #2

While I was writing the article about NAS, thought flashed across my mind – can we avoid writing to the temporary drive and reduce the load ONCE more?

Yes, we can. Mochilas Kanken No.2 I know that RAM Drives are also out of fashion, but here one will come handy. It’s the shame that support for it isn’t included in the system already, so with little googling I found this: http://www.ltr-data.se/opencode.html/#ImDisk

I installed it on the testing server, re-configured the application to use new temporary folder, and from now on, it runs so smooth I can’t hear it anymore :)

Some technical stuff:

  • in this instance, I am currently recording 62 channels and cumulative rate for it is around 5 megabit/second
  • my files have duration of 5 minutes, which means that recorded chunks are closed and moved to permanent storage every 5 minutes
  • during those 5 minutes, each file will grow so much that the whole content for those 5 minutes won’t get over 200megabytes
  • I created 512 megabyte ram drive, just to be safe

Conclusion

Take care of your hard drive, and don’t dismiss old-techs such as RAM Drives just yet.

If I was about to implement this on an application level, I would have to spend a great deal of time, and some media types won’t even be possible to implement – Windows Media for example, writes to disk or to other places if you employ magic…

Having NAS is great (or is it?)

Tuesday, May 17th, 2011

During the years I had many deliberations over the fact if either NAS would be used or it wouldn’t be used for the video archives created by video logger system such as VideoPhill Recorder. air max pas cher At first, I was firm believer in one methodology, then completely turned my side to the other, and now, … Well, read on, and I’ll take you through it.

Stakes

On the one side of the stake set we have actual requirements, and on the other there are considerations. Actual requirement are sometimes hard to pinpoint at first, but they always come out sooner or later.

So, let me list possible requirements that might be in effect here…

Common low level requirements

Storage for video recording (for logging purposes) needs to have following abilities:

  • low but constant and sequential write rate – data rate for 4 channels are as low as 5mbit per second (500kbytes/sec) but is CONSTANT and SEQUENTIAL – there won’t be much stress for the hard drive because of constant seeking
  • high durability over time – what gets written once, has to be there. Nike Air Max TN Homme It should survive single drive failure
  • reading isn’t common, but when done, it has to be sustainable, but again at low data-rate and great predictability (it usually is sequential)
From everything above, I can guess that any data storage expert would read RAID 5 and won’t allow you to create anything else for the video archive storage.

Archive duration scalability

The archive duration is directly proportional with the hard drive space that is available. To determine what kind of hard drive space you need for your first installation, you can use on-line hard drive size estimator calculator that I created right for this blog.

If you plan extend the duration of your archive one day, you have this requirement, and you have to plan for it. Having the storage at one place can simplify the adding of the drive space, but can also completely block it.

Let’s say that you have the archive of 40 channels that span 92 days (3 months). And let’s say that you decided to use 1mbit video with 128 kbit audio for the archive. By using the calculator above, you’ll find out that you have 42 terabytes of storage already in place. Even at this date, that kind of storage set in one place is kind-of-a challenge to build.

If you have already invested in 42 TB storage system, and have foresight to plan for an upgrade to say its double size for it, you are in luck. But, say that after a few more months (just 3) your management decides to expand the requirement to 12 months of storage. Wow. nike air max 90 pas cher Now, you have to have 127 TB total. If the current system will hold that much drive space, again you are in luck, however – say it doesn’t. Nike Air Max 2016 Heren blauw Your options are:

  • add one more to the chain
  • create a bigger unit, copy everything to it, scrape the current one
I’ll stop my train of thought here, and leave you only with few things to think about before I go on with other requirements: who needs used 82 TB system (if you want to sell it), do you know how much it is to COPY 82 TB even at extreme network speeds, adding one more will break the ‘all in one place’ requirement, …

Having it all in one place (i.e. for web publishing)

If you need everything in one place for the publishing, then this is a solid requirement. Scarpe Adidas Web server will have the content on its local hard drives, and it will publish it smoothly.

But, is that really a requirement? I must admit that I didn’t see web server that properly served files from the network locations (despite the thing that there are option to do that), but I’m sure that IIS and Windows Server gurus will be able to shut me down and say that this is normal thing that is done routinely. So, if we know that each channel recorder has its own directory ANYWAY, what’s the use of having

c:\archive\channel1
c:\archive\channel2
c:\archive\channel3

instead of

\\rec1\channel1
\\rec1\channel2
\\rec2\channel3

Reducing single point of failure

This requirement is very common, and having NAS system as a ‘point’ it brings us that having the NAS leads us to having single point of failure. I understand that there are multiple redundancies that could be installed into the system, such as RAID 5, or obscene configurations such as RAID 1+5. Note: for later, it seems that the article author has a same opinion on it as me:

Recommended Uses: Critical applications requiring very high fault tolerance. In my opinion, if you get to the point of needing this much fault tolerance this badly, you should be looking beyond RAID to remote mirroring, clustering or other redundant server setups; RAID 10 provides most of the benefits with better performance and lower cost. Not widely implemented.

In my words: if you need such system, it’s better to have recording drives distributed on each recording machine, have RAID 5 there, and additionally have NAS (or some other form of storage) to DUPLICATE everything.

Bandwidth issues

At a configuration with 4 channels recorded at one machine, and with above mentioned data rate for video and audio, each machine will produce 5 megabit of content every second. Roughly, that is .5 megabyte. Even ZIP drive could almost handle that. However, if you have 10 times that (for 10 recorders) and have central storage for the whole bunch of channels, that is 5 megabytes of data at a constant rate that never stops.

Consider that central storage in question is NAS has hard drives configured in RAID 5. That means that it will have to receive, calculate parity for, move the drive heads, write to drive, … It will be very busy NAS, and with everything else in mind, it won’t have a second of a break. Add to that occasional reading of the content from the archive diggers, and you’ll soon figure out that the NAS will have to take it all itself.

On the other hand, archive access applications such as VideoPhill Player doesn’t have anything against having the channels on different recorder machines.

Conclusion (for bandwidth issues) – having each recorder machine handle both encoding and storage for 4 channels will reduce single point of stress for both recording and the archive access.

Overall…

Having dumped my intuition in this few paragraphs, I hope that I presented case that is strong enough against having NAS for video logger/archive storage. Nike Air Max Goedkoop Again, everything said is from my experience on the subject, and I’m no storage expert who will talk petabytes, just a simple consultant trying to get my clients best bang for the buck.

And then, I got in… (story of data visualization)

Sunday, May 15th, 2011

How to see the data?

If the data is numeric, and it represents some series, it will be mostly represented with a graph of some sort. There are hundredths types of graphs available, and they all have some purpose, otherwise they would not exist.

However, for some special occasions, you have to see different kind of data.

The problem (this particular instance)

Since I am developing a internet media streaming CAPTURE and ARCHIVE application (StreamSink) I am also continuously testing it on one of my servers. I am adding channels, removing them, stopping the server, sometimes something goes wrong and the whole thing freezes or crashes, so the archive I have is rather heterogeneous in quality.

Let me go through the operational view – the mere GUI of the StreamSink, so I can present some problems and solutions so far.

StreamSink

Several things were important to the operator of the software that had to be present on the main (status) screen. For example:

  • whole list of channels should be visible
  • channel status should be visible at first glance
  • I am interested what happened to the system recently
  • I need to know the status of my connection
  • it would be good to know how many disk space is available

I could dwell on it but the main point of this post is something else.

The problem here is that I had to create PlayKontrol report for a demonstration purpose (for them: http://ihg.hr/), that would scan 7 days of the archive (multiple channels, of course), and produce the reports (playlists) for 300 songs.

So the problem is: to

find, in the archive that is damaged in various ways, 7 days of continuous archive that spans multiple channels.

The solution (prelude)

Since I am kind of explorer by nature, I wasn’t inclined to use a solution that would present raw data as an answer, but was into thinking about seeing the data and determining the period and channels ‘visually’.

StreamSink has a integrated feature that is called ‘archive report’, that has data similar to what I need, but with it I would only get limited information. You can see the report here:

StreamSink Archive Report

Most useful info on the report in this particular situation would be the graph on the right side of the report. Nike Air Max 2016 Heren wit Let me explain…

For each day StreamSink is able to record up to 24 hours of media. Fjallraven d’Occasion Due to network situations, it sometimes is less then 24 hours, and I decided that I would present that number in the form of percentage that archive is covered for the day. As you can see from the report, that percentage is shown for the whole archive lifetime, for last month, last week and last 24 hours.

Also, it is shown in the form of graph, where on the leftmost part of the graph is the current day, and as we go to the right, we sink onto the past, having divider lines at each 7 days. air max 1 pas cher Nice, eh? :)

But, as nice as that report is, I can’t read what 7 days and what channels are to be scanned – I have to find another way in.

Solution (at last)

For this one, I picked something that I learned from the above mentioned report. Cheap Fjallraven Kanken Outlet That was:

  • I will have a channel list
  • I will have some sort of calendar
  • I have to see how much is covered for the archive for each day

Also I decided to show each day as a cell in a table-style matrix, where rows would be occupied by channels, and columns will be days. Time flow was inverted here, so left is past, and right is the present.

Whole thing looks like this:

Archive Digger

Same thing little zoomed in:

Archive Digger Detail

Note: green is the color for the days that have 90% or more archive covered.

At last, you can see from the both pictures that much of the data is revealed at the first glance. For example, 0 means that there were no archive that day at all. Nike Max Shoes UK Numbers below 90 suggest that either it was some problem with the channel that day, or StreamSink was either started or stopped in the middle of the day.

I could even color-code that information on the chart – but the utility will be expanded further only if there’ll be demand for it, since I know what I needed to know, from it.

BTW, I don’t want to brag here, but to code that utility it took 2-3 hours of thinking and coding, and almost no debugging.