[3DS][SSL][SDK] Unchecked number of audio channels in Mobiclip SDK leads to RCE in eShop movie player

Summary

#Affected Systems

• Platform: (New) Nintendo 3DS
• Region: ALL
• System version: 11.13 (latest at the time of writing)

#Description The Mobiclip SDK used for parsing moflex videos does not check the number of audio channels in an audio stream. This leads to a miscalculation of free space remaining in a heap buffer, and then a buffer overflow.

#Vulnerability The mw::mo::helper::PcmAudioPresentation::GetNextAudioDataPtr function is used to calculate the address where the incoming audio data should be copied before getting processed:

void* GetNextAudioDataPtr(audio_presentation* ap, uint32_t data_size) {
    int64_t offset = ap->bufferOffset;
    int64_t already_played = GetTotalSamplePlayed(ap->soundId); //this is calculated like this: (previous_data_length / (nb_channels*2))
    int64_t already_played_bytes = already_played << ap->audioStreamInfo->nbChannels;

    uint32_t free_space = ap->bufferSize - (uint32_t)(offset - already_played_bytes);
    uint32_t remaining_space = ap->bufferSize - offset;

    if(remaining_space < data_size) {
        offset = 0;
        free_space -= remaining_space; 
    }
    if(free_space < data_size)
        return NULL; 
    return ap->buffer + offset;
}

We can see that to get the number of already used bytes in the audio buffer it first gets the number of "audio units" already used (number of shorts in PCM16 format). This is done by calling the GetTotalSamplePlayed function, which relies on the GetSampleLength function. Thus, the already_played value is calculated like this for PCM16 data: previous_data_length / (nb_channels*2). This is the number of PCM16 data units that have already been processed. Then to get the number of bytes already processed, already_played_bytes is calculated based on already_played like this: already_played_bytes = already_played << nb_channels, which is equivalent to already_played_bytes = already_played * pow(2, nb_channels). When the number of channels is 1 or 2, this is correct because 2 * nb_channels = pow(2, nb_channels). However the number of channels is provided by video files and is not checked, thus it is possible to set a number of channels greater than 2 (up to 256). When the number of channels is greater than 2, pow(2, nb_channels) > 2 * nb_channels, thus the calculation of free_space becomes incorrect and can be far greater than the real free space remaining. One can for example set the number of already processed data and the number of channels such that already_played_bytes = 0x8XXXXXXX which is enough to totally bypass the size check.

Impact

Because of #894922, anyone can exploit this vulnerability and get remote code execution in usermode under the eShop application. Moreover, this could potentially(?) be exploited without any SSL flaw by someone able to submit games to the eShop catalog, however since I don't know the validation process and checks performed, I cannot ensure this is possible. Note that one can exploit this vulnerability without any noticeable change to the eShop application behavior. This means the user has no way to know something bad is happening before launching the video playback.