JAVA AND DSP INTEGRATION FOR WIRELESS MULTIMEDIA

Acceptance of the Java programming language ensures its use in bringing multimedia services to the wireless environment. A roadmap in the form of a data model shows how it may be accomplished.

By Jason Brewer and Marion Lineberry

While many aspects of third-generation wireless systems are still undecided, it is certain that they will boast multimedia capabilities. By any estimate, this will be a major task and will require large development efforts in the next few years. Multimedia applications such as speech recognition, map-based navigation video, and Internet access all require interfaces today found only in portable and desktop microcomputers. Programming languages such as Java, along with programmable DSPs, will play a large role in the integration of these applications in wireless networks as well.

Constructing a model is helpful in illustrating how DSP-implemented function blocks can be integrated into Java applications. While this model is not necessarily the ideal solution for all DSP applications, it is representative of how solutions that communicate with DSPs through task I/O channels will look. In constructing the model, it is assumed that the DSP kernel handles communication between the DSP and the host processor through I/O channels and task management. Functions performed on the DSP must receive their input from the application host and be initiated after this input has been set up. This is accomplished by identifying each function with a respective task wrapper implemented on the DSP.

This wrapper reads the function input from the input channel, allocates any memory necessary for the output, calls the function with the input, and writes the output to the output channel when the function call has completed. Each task has an associated control structure that describes its input and output so that the necessary I/O channels can be set up before the function is called.

Scenarios for Java and DSP integration

Due to its interpretative nature and implementation requirements, Java currently isn’t suited to perform processing-intensive tasks, such as media compression or speech interpretation. Yet, many applications require the execution of these tasks. Because of this, some Java libraries are defined as abstract classes that can be implemented in native host code, yet conform to a standard Java application programming interface (API) at the top level so applications can still incorporate their functionality.

One scenario for implementing process-intensive tasks is to offload them onto a coprocessor, such as a DSP, so that such functionality is implemented in DSP code and called from Java according to the publicly defined interface for such a task. For example, an MPEG media player might reside locally in a media package specific to the DSP and be loaded onto the DSP when needed, thus freeing the host processor to perform other duties. It would be beneficial to media player developers if there was an easy way to incorporate such solutions into Java. Third-party Java application developers could take advantage of these implementations since they would have a well-defined interface.

Another scenario is to allow Java applications to take advantage of existing DSP libraries by providing a standard way to call them from Java. This might be useful for application prototyping or developing novel applications. This is not much different than the first scenario, except that there may be no standard public interface for the DSP task being developed. Because of this, the DSP developer would have to create a new interface. The DSP code would then only be useable for applications aware of this interface.

A third scenario is to use the DSP to accelerate individual Java methods when it can perform tasks faster than the local host. The idea is to call a DSP implementation of a method rather than the local host or Java version to provide accelerated performance of method execution. Since this article concentrates on DSP application interface issues, this scenario will not be discussed.

The data flow model

Java provides a means of incorporating native host code with a C or C++ interface into Java applications through a defined Java native interface. Thus, DSP libraries could be called from Java if a developer wished to wrap them in a C interface native to the host, which then called the DSP. It would be ideal if there was a standard Java API for instructing the host to call DSP library functions, so that a developer didn’t have to worry about this native interface. This would insulate the developer from the method the client host platform uses to interface to the DSP.

The model presented here is a way of incorporating DSP functionality into Java by wrapping the functionality into a data flow object that has associated input and output channel objects for communication. The data flow object wraps a DSP function task and provides a channel I/O interface for connecting the task to other data flow or Java objects.

In Figure 1 , a task wrapper for a DSP function shows how the DSP kernel must call a DSP function. The function is enveloped in a task wrapper that is in charge of performing the I/O operations between the DSP and host. Assuming the use of a kernel based on tasks and I/O channels, all of the above blocks exist on the DSP, independently of the model used to access them from Java. A data flow model fits this representation very well.

The data flow object becomes the Java representation of the DSP task wrapper. When a Java data flow object is created, the object constructor checks to see if the associated DSP task for that object can be created. This can be accomplished through a DSP API call to negotiate with the DSP kernel for DSP resources. Once the task is initialized, input and output channel objects are created that are attached to the input and output channels of the DSP task. In Figure 2 , the Java representation of a data flow object with input and output channel objects is shown. This implementation can in turn be wrapped by a standard Java interface, such as a media player API, to insulate the end application from the DSP I/O specifics.

The data flow model is not required for accessing DSP code from Java, but it is a natural extension of how DSP functions might be wrapped into tasks on the DSP. Most of the overhead is involved in the initial setup of the data flow object. Once the object has been created, most applications only need to push and pull data through the I/O channels. For efficiency, DSP functionality should be as coarse as possible with respect to the performed task. This should minimize the amount of I/O that needs to be done through the Java data flow object. Thus, with the data flow model, it makes more sense to model DSP functionality, such as a complete audio encoder, as a single object, rather than implementing it with individual function blocks connected to multiple data flow objects. A multiple object implementation might be useful for prototyping or developing custom DSP algorithms, but will suffer in performance compared to the same functionality implemented as a single object.

Implementation details

This section provides an overview of how a Java application might use the data flow model to incorporate DSP library code into its functionality. First, assume a DSP library function exists to encode a buffer of audio data:

Encode(char *inbuffer, char

*outbuffer, int rate);

This function has an input array for the raw audio, an output array for the coded audio, and a rate parameter for the coding method. In order for this function to be called from a separate host through the DSP kernel, a task wrapper is necessary. This includes the control structure for specifying the input and output channels as well as the functionality to read and write the input and output data and allocate any necessary memory for the task:

TaskControlStruct{In, Out};

TaskEncode(){

ReadInput(In, rate);

ReadInput(In, inbuffer,

isize);

alloc(outbuffer);

Encode(inbuffer, outbuffer,

rate);

WriteOutput(Out, outbuffer,

osize);

}

This forms the wrapper implementation on the DSP. Before the task can be started from a Java program, it is necessary to substantiate a class that implements the DSP host-interface API for host-to-DSP task communication. In the discussion below, this is the TaskManager Class. The rest of the code below is in Java:

TaskManager TM =

system.getTaskManager();

Next, the program must request that the TaskManager initialize the encoding task. Assuming the resources are available on the DSP to execute the task, a task ID might be returned. This could be implemented in several ways. The example below assumes there is some type of DSP task registry indexed by task name. The DSP device driver would most likely track this information, and applications would initiate task negotiation with the DSP kernel through the DSP host-interface API:

int tskID =

TM.createTask(“Encode”);

Next, the I/O channels for communicating with the task need to be created and associated with the newly created task ID:

InputChannel iC = new

InputChannel();

OutputChannel oC = new

OutputChannel();

TM.connectInput(tskID, iC);

TM.connectOutput(tskID, oC);

The task can now be started, and the application can write the necessary input and read the associated output. For a streaming function such as an encoder, this would probably be done in independent threads:

TM.start(tskID);

iC.writeIntParam(rate);

iC.writeByteParam(inbuffer[]);

oC.readBytes(outBuffer[]);

This is just an example of how a data flow model might be implemented. A real implementation would be dependent on the DSP kernel implementation and the DSP host-interface API calls. Additionally, it is not necessary for developers to implement this level of detail. A Java encoding package could be developed from the above implementation that insulated third party developers from the I/O specifics. For example,

codec = new Encoder(rate);

codec.encode(inbuffer[],

outbuffer[]);

This adds another implementation level, but could be valuable if the ease of implementation results in more developers being able to use the wrapped DSP functionality. A block diagram of the architecture is shown in Figure 3 . The Java application uses the task I/O classes to create tasks, read input, and write output from those tasks. The task I/O classes achieve this functionality through calls to the underlying DSP API. This API could be implemented in Java, native host code, or both.

Distributed processing

The previously described scenario only contains a single data-flow object. Multiple data-flow objects can be chained together by connecting their input and output channels. This could be done with library functions on a single DSP, or it could be used to create a distributed processing system across multiple DSPs or host systems.

For distributed processing, the local application connects to another host’s TaskManager and negotiates for resources in the same manner it does for local DSP resource use. Input and output channels then span between multiple hosts across a network or other medium. An example of distributed processing for speech recognition is shown in Figure 4 .

Here, an application has set up a local task to take direct speech input to the DSP and perform “phoneme” recognition. It has directed the output of the task to a remote server host to perform the actual recognition. This would be valuable in scenarios where the client has limited memory for vocabulary and grammar. The setup details of this distributed task are not shown in the figure.

Presumably, before the application begins the speech recognition process, it has negotiated with the server for processing resources and the necessary network connections have been established. Once the server has performed the recognition, it returns the resulting speech to the client application. The difference between the distributed and local scenarios is the source and destination of the task data. Also, the server host has enhanced task I/O classes that negotiate for resources with the local DSP kernel on behalf of remote hosts.

Virtualizing I/O channels

It is not necessary for data flow objects to move input and output data from Java. The Java input and output channel objects could actually be virtual representations of I/O streams that come from other hardware devices or software components. The task I/O setup can just use the Java I/O channel objects to inform hardware device drivers where to direct their input and output. This is shown in the previous example. The Java application would probably have an abstract representation of an object for speech input. Although the application never actually touches the speech data, it interfaces with a microphone device driver to start recording and directing the resulting raw audio data to some component that can process it — in this case, the DSP.

JavaSoft InfoBus specification

JavaSoft is promoting a new specification called InfoBus that describes a way for software components such as Java Beans to dynamically exchange data. The InfoBus allows these components to plug into a common data bus and defines interfaces for communication between individual components. The information is exchanged as DataItem objects that implement a standard DataItem “access” interface. Components can retrieve and navigate a DataItem’s content through this interface. This interface might also be used to connect DSP software components, possibly in the data flow model described.

The intent of the InfoBus is to impose structure on the data to make the data comprehensible to the widest audience of consumer components. Input streams of multimedia content might not lend themselves to such formatting. The specification defines interfaces for standard data types with ArrayAccess and ImmediateAccess, but there is no specification for a StreamInterface. The InfoBus specification recommends that streams of data be handled via the transferable mechanism, according to the AWT clipboard specifications.

This is an interface for requesting representations of data at various levels of detail. However, it is unclear in what context it might be used to transfer live data streams. Regardless of how stream transfer is implemented, the InfoBus might be minimally useful for messaging between components to set up and direct stream data independently of the InfoBus. An example scenario is given in Figure 5 .

Here, three independent components exist in a system, possibly created by a single or multiple parent applications. The third-party component might query the InfoBus for a source of media streams. The network stream component, which responds to such queries, replies via the InfoBus that it has the capability to provide notifications of incoming streams. When the third-party component receives a notification of a stream that it is interested in, it might direct the network stream component to send a network media stream to the media component, which it also queries to startup, for playout. In this scenario, the third-party component might be acting as a DataController object for the InfoBus. This interface is not currently specified, but is described as providing intelligent traffic control on the bus.

This scenario could easily be accomplished in a single application without the InfoBus. However, the InfoBus allows a more autonomous distribution of components so that they become useful to many applications. Since this specification is still under development, it is unclear what its final benefits may be, but it appears to be useful for developing distributed applications and communicating more easily between independent Java Bean components in a standard manner.

Figure 6

No panacea

As stated earlier, this data flow model is not the panacea for every application, but merely representative. Large tasks implemented in a single block might not be traditionally thought of as fitting a data flow model, but it is still useful to set up the input and output for such tasks in the same manner as for a data flow object. Performance issues will arise when chaining multiple objects together, but such architectures can still be useful for prototyping and algorithmic experimentation. Having tasks represented as Java objects is very appealing for such development scenarios. This model can also be useful in implementing distributed tasks.

It is too soon to tell whether InfoBus interfaces will be useful for DSP tasks. It might be possible to implement the communication between tasks in the data flow model using standard-class APIs provided in the InfoBus specification. To fit into this architecture, DSP tasks could be wrapped in Java Bean components. This is inviting for development ease and downloading of tasks, but further investigation is needed to compare the overhead required between the described data flow model and one based on InfoBus and Java Beans.

Finally, any implementation model will be dependent on the underlying API for host-to-DSP communication (See the sidebar on “ The Importance of the API,” p. 27 ) . If the I/O channel implementation does not fit the underlying communication mechanism, a different model that is based on the underlying platform is necessary. Native implementations will always be an option. Advanced developers can code to their individual platform and make their own host-to-DSP interface based on the underlying kernels.

Any other implementation based on a standard Java interface, such as the data flow model discussed here, risks some inefficiency. Many of the advanced processing tasks that the Java-DSP platform seeks to enable will not fit such a generic interface model. Rather, the purpose of such a model is to simplify application programming for this platform. This is the main advantage of the data flow model.

Jason Brewer is a member of the technical staff with the DSP solutions R&D center of Texas Instruments in Dallas, Texas. He received his MSEE and BSEE from Texas A&M University.

Marion Lineberry is a senior member of the technical staff with the wireless communications business unit of TI in Dallas, Texas. He received his MS and BS in math and computer science from the University of Illinois. He can be reached at lineberry@ti.com.