Introduction

Introduction to ESB3034 - the ew-cmaf-ingest service

Ingestion of Segmented Live ABR Content

The traditional way of ingesting live ABR content uses MPEG-2 TS streams, preferably in ATS format. The TS-packets are transported over plain UDP, over UDP with RTP headers and possible FEC, or using SRT for reliable transport. Such streams can be received by Agile TV’s live-ingest component ESB3003, which segments the content for further manipulation and distribution via AgileTV Origin’s other components.

However, given that the end-goal is to distribute a segmented stream, an alternative is that the segmentation is done already by the encoder, and that such segments are ingested to AgileTV Origin. For a long-time, the formats for this type of ingest have been under-specified, but with the DASH-IF CMAF Ingest specification, there is now a common ground for interoperability. The AgileTV Origin component ew-cmaf-ingest supports this format, and receives streams of segments. It buffers, reformats, and encrypts the content but leave the segmentation unchanged, except for alignment to CMAF’s internal numbering and timing, and adding additional metadata if configured.

Ingest format and URLs

The CMAF Ingest specification defines two interfaces. ew-cmaf-ingest currently only supports Interface-1, which is a stream of segments starting with an init segment for each variant. Interface-1 does not require a DASH manifest, but relies on metadata inside the segment tracks to find language, bitrate, role etc.

For cases where the embedded metadata is lacking or not correct, ew-cmaf-ingest allows for configuring the values. The configuration can also be used to override the incoming values.

The ew-cmaf-ingest services receives CMAF segments being pushed to it using either HTTP PUT or HTTP POST. To avoid un-authorized content to be ingested, the content should be pushed over HTTPS and use basic auth with preconfigured username and password for each channel.

The receiver parses the URLs of the pushed initialization and media segments. They should be in one of two forms, where the first form as the same URL for all segments of a specific channel and track:

/<prefix>/<channelName>/Streams(<trackName>.cmf?)
/<prefix>/<channelName>/<trackName>/<segmentID>.cmf?

Here,

  • <prefix> is a configured URL prefix for all channels
  • <channelName> is the name of the channel
  • <trackName> is the name of a specific track
  • .cmf? is a segment-type dependent file extension .cmfv, .cmfa, .cmft, .cmfm for video, audio, text, and metadata, respectively
  • segmentID is a name of a segment. This actual name is not used, since the receiver does generally not understand the name if a Manifest is not available

In addition, the source can send a DASH manifest to a URL like:

/<prefix>/<channelName>/<trackName>/<manifestName>.mpd

Manifests will not be parsed, but just stored as <storage>/asset/received.mpd.

HTTP DELETE requests are accepted, but not used, since there is a built in max buffer time resulting in a maximum number of segments stored and being available for further processing.

Fragmented segments and low-latency streaming

For efficiency, it is preferred that Content-Length headers are used in the HTTP POST and PUT requests for segments. This should be possible for all init segments, but not for chunked low-latency segments where the chunks are generated and pushed using Chunked Transfer Encoding before the full segment is finished.

ew-cmaf-ingest will keep the CMAF chunks of the incoming segments and propagate them to the rest of stream-builder as soon as all tracks have received the corresponding chunks. It is therefore important that all low-latency media tracks use the same chunk duration.

How to run

Installation

The ew-cmaf-ingest service is distributed as a CentOS/Red Hat RPM and is available as ew-cmaf-ingest.service when installed. It works on Red Hat 8 and compatible versions.

To install version x.y.z, the rpm command can be used as follows:

dnf install ./ew-cmaf-ingest-x.y.z-1.el8.x86_64.rpm

The service can be controlled with the standard systemctl commands:

systemctl start ew-cmaf-ingest
systemctl stop ew-cmaf-ingest
systemctl status ew-cmaf-ingest

Logs from the service can be seen using journalctl -u ew-cmaf-ingest.

The unit file is available in /usr/lib/systemd/system/ew-cmaf-ingest.service.

When run as a service, ew-cmaf-ingest reads its initial configuration from /etc/edgeware/ew-cmaf-ingest/config.json. To uninstall the service, do:

dnf remove ew-cmaf-ingest

When uninstalling it is recommended to first stop the service.

Configuration

For configuration, see the Configuration section.

Segment processing details

Segments are processed in parallel and adopted to align with AgileTV Origin’s mezzanine format ESF. That format is very well aligned with the recommendations in the CMAF Ingest specification:

  1. The reference time is set to 1970-01-01T00:00:00Z by shifting the time of the input segments with an appropriate amount. If the CreationTime in the [mvhd] is set to the start of a full year, that value is used to establish an exact relation between input and output timestamps.
  2. The media segments sequence numbers are parsed and used to define a fixed offset between input and output sequence numbers. The output sequence numbers (used in the rest of the AgileTV Origin) corresponds to startNumber=0 at 1970-01-01T00:00:00Z. It is possible to configure an initial offset for faster startup.
  3. Language code is extracted from the [mdhd] or [elng] boxes unless configured for the track
  4. Bandwidth is extracted from a [Btrt] box. If not present, it is estimated from the duration of the first segment. It can also be set in the configuration.
  5. Roles are extracted from [trak.udta.kind] boxes, but can also be configured.
  6. The output segment generation does not start until two consecutive video segments of the same track with identical durations have arrived. That duration is then used as the reference for the full channel and is not allowed to vary.
  7. Metadata tracks are received and SCTE-35 information is extracted. The information is currently not further used.

Supported Codecs

  • video: AVC/H.264, HEVC/H.265
  • audio: AAC-LC, HE-AACv1, HE-AACv2, AC-3, EC-3
  • subtitles: wvtt, stpp

Note that all stpp-subtitles will be transformed into wvtt subtitles for internal storage.