发表于 | 分类于 |
字数统计: 444字 | 阅读时长 ≈ 3分钟 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
import android.media.MediaCodecList;

import java.util.HashMap;
import java.util.Map;

public class VideoSupportUitl {

    private static Map<String, String> codecMap = new HashMap<>();
    static {
        codecMap.put("h264", "video/avc");
    }

    public static String findVideoCodecName(String ffcodename){
        if(codecMap.containsKey(ffcodename))
        {
            return codecMap.get(ffcodename);
        }
        return "";
    }

    public static boolean isSupportCodec(String ffcodecname){
        boolean supportvideo = false;
        int count = MediaCodecList.getCodecCount();
        for(int i = 0; i < count; i++)
        {
            String[] tyeps = MediaCodecList.getCodecInfoAt(i).getSupportedTypes();
            for(int j = 0; j < tyeps.length; j++)
            {
                if(tyeps[j].equals(findVideoCodecName(ffcodecname)))
                {
                    supportvideo = true;
                    break;
                }
            }
            if(supportvideo)
            {
                break;
            }
        }
        return supportvideo;
    }
}

具体的类型对应关系可以查看相关文档，这里在Android源码MediaCodec.createDecoderByType()里面有一些相关的对应支持类型。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

/**
    * Instantiate the preferred decoder supporting input data of the given mime type.
    *
    * The following is a partial list of defined mime types and their semantics:
    * <ul>
    * <li>"video/x-vnd.on2.vp8" - VP8 video (i.e. video in .webm)
    * <li>"video/x-vnd.on2.vp9" - VP9 video (i.e. video in .webm)
    * <li>"video/avc" - H.264/AVC video
    * <li>"video/hevc" - H.265/HEVC video
    * <li>"video/mp4v-es" - MPEG4 video
    * <li>"video/3gpp" - H.263 video
    * <li>"audio/3gpp" - AMR narrowband audio
    * <li>"audio/amr-wb" - AMR wideband audio
    * <li>"audio/mpeg" - MPEG1/2 audio layer III
    * <li>"audio/mp4a-latm" - AAC audio (note, this is raw AAC packets, not packaged in LATM!)
    * <li>"audio/vorbis" - vorbis audio
    * <li>"audio/g711-alaw" - G.711 alaw audio
    * <li>"audio/g711-mlaw" - G.711 ulaw audio
    * </ul>
    *
    * <strong>Note:</strong> It is preferred to use {@link MediaCodecList#findDecoderForFormat}
    * and {@link #createByCodecName} to ensure that the resulting codec can handle a
    * given format.
    *
    * @param type The mime type of the input data.
    * @throws IOException if the codec cannot be created.
    * @throws IllegalArgumentException if type is not a valid mime type.
    * @throws NullPointerException if type is null.
    */
   @NonNull
   public static MediaCodec createDecoderByType(@NonNull String type)
           throws IOException {
       return new MediaCodec(type, true /* nameIsType */, false /* encoder */);
   }

在ffmpeg里面

1
((const AVCodec*)(avCodecContext->codec))->name;

即可拿到name，然后jni交互调用即可。

发表于 | 分类于 |
字数统计: 314字 | 阅读时长 ≈ 1分钟

FFmpeg解码获得的AVPacket只包含视频压缩数据，并没有包含相关的解码信息
（比如：h264的sps pps头信息，AAC的adts头信息），没有这些编码头信息解
码器（MediaCodec）是识别不到不能解码的。在FFmpeg中，这些头信息是保存
在解码器上下文（AVCodecContext）的extradata中的，所以我们需要为每一种
格式的视频添加相应的解码头信息，这样解码器（MediaCodec）才能正确解析
每一个AVPacket里的视频数据。

主要使用的类AVBitStreamFilter

添加过程:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

const AVBitStreamFilter *absFilter = NULL;
AVBSFContext *absCtx = NULL;
AVCodecParameters *codecpar = NULL;

//1. 找到相应解码器的过滤器
if(strcasecmp(codecName, "h264") == 0){
    absFilter = av_bsf_get_by_name("h264_mp4toannexb");
}else if(strcasecmp(codecName, "h265") == 0){
    absFilter = av_bsf_get_by_name("hevc_mp4toannexb");
}

//2.过滤器分配内存
av_bsf_alloc(absFilter,absCtx)

//3. 添加解码器属性
if(pFormatCtx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO){
    codecpar = pFormatCtx->streams[i]->codecpar;
}
avcodec_parameters_copy(absCtx->par_in, codecpar);

//4. 初始化过滤器上下文
av_bsf_init(absCtx);

//5. AVPacket处理
if(av_bsf_send_packet(absCtx, avPacket) != 0){
    av_packet_free(&avPacket);
    av_free(avPacket);
    avPacket = NULL;
    continue;
}
while(av_bsf_receive_packet(absCtx, avPacket) == 0){
    LOGE("开始解码");
    av_packet_free(&avPacket);
    av_free(avPacket);
    continue;
}
avPacket = NULL;

//6. 释放资源
av_bsf_free(&absCtx);
absCtx = NULL;

ffmpeg流过滤器列表

发表于 | 分类于 |
字数统计: 1,128字 | 阅读时长 ≈ 5分钟
视频播放思路

和播放音频一样，采用生产者消费者模型。AvPacket入队，然后AvPacket出队伍解码。

视频解码渲染

软解码：如果解码之后的数据格式是AV_PIX_FMT_YUV420P直接使用采用OpenGLES渲染，如果不是AV_PIX_FMT_YUV420P采用sws_scale转为AV_PIX_FMT_YUV420P在采用OpenGLES渲染。将YUV数据转换RGB的操作放在OpenGLES里面，使用GPU提升效率。软解码容易造成容易造成音视频不同步。
硬解码：在解码之前判断是否支持硬解码，如果支持硬解码就直接通过ffmpeg处理视频数据H264 H265等，为其加上头信息，然后硬解码交其OpenGLES渲染。

音视频同步问题
  1. 音频线性播放，视频同步到音频上。
  2. 视频线性播放，音频同步到视频上。
  3. 用一个外部线性时间，音频和视频都同步到这个外部时间上。

由于人们对声音更敏感，视频画面的一会儿快一会儿慢是察觉不出来的。而
声音的节奏变化是很容易察觉的。所以我们这里采用第一种方式来同步音视频。
这里需要计算当前视频帧的播放时间和当前音频的播放时间来进行比较，然后计算出睡眠时间来让视频不渲染还是延迟渲染,保持音视频尽量同步。

音视频同步相关计算
  • 计算当前视频帧播放的时间
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
double clock = 0;

if(pFormatCtx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO){
    time_base = pFormatCtx->streams[i]->time_base;
}

//...

 double pts = av_frame_get_best_effort_timestamp(avFrame);
    if(pts == AV_NOPTS_VALUE)
    {
        pts = 0;
    }
    pts *= av_q2d(time_base);

    if(pts > 0)
    {
        clock = pts;
    }
  • 计算音视频播放时间差值 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

//如果>0表示音频播放在前，视频渲染慢了，需要加速渲染  <0表示音频播放在后，视频渲染快了，需要延迟渲染
double getFrameDiffTime(AVFrame *avFrame) {
    double pts = av_frame_get_best_effort_timestamp(avFrame);
    if(pts == AV_NOPTS_VALUE)
    {
        pts = 0;
    }
    pts *= av_q2d(time_base);

    if(pts > 0)
    {
        clock = pts;
    }

    double diff = audio->clock - clock;
    return diff;
}
  • 计算渲染睡眠时间
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
 //延时时间 单位秒
double delayTime = 0;
//默认的延时时间 通过当前帧的AVRational计算fps所得  单位秒
double defaultDelayTime = 0.04;

if(pFormatCtx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_VIDEO){
    int num = pFormatCtx->streams[i]->avg_frame_rate.num;
    int den = pFormatCtx->streams[i]->avg_frame_rate.den;
    if(num != 0 && den != 0)
    {
        int fps = num / den;//[25 / 1]
        defaultDelayTime = 1.0 / fps;
    }
}

double getDelayTime(double diff) {
    //如果音频的播放时间超过了30ms 视频需要加速渲染 慢慢的缩小睡眠时间 达到平缓的效果
    if (diff > 0.003) {
        delayTime = delayTime * 2 / 3;

        if (delayTime < defaultDelayTime / 2) {
            delayTime = defaultDelayTime * 2 / 3;
        } else if (delayTime > defaultDelayTime * 2) {
            delayTime = defaultDelayTime * 2;
        }
    } else if (diff < -0.003) { //如果音频的播放时间慢了30ms  视频需要延迟渲染
        delayTime = delayTime * 3 / 2;
        if (delayTime < defaultDelayTime / 2) {
            delayTime = defaultDelayTime * 2 / 3;
        } else if (delayTime > defaultDelayTime * 2) {
            delayTime = defaultDelayTime * 2;
        }
    } else if (diff == 0.003) {

    }
    if (diff >= 0.5) {
        delayTime = 0;
    } else if (diff <= -0.5) {
        delayTime = defaultDelayTime * 2;
    }

    if (fabs(diff) >= 10) {
        delayTime = defaultDelayTime;
    }
    return delayTime;
}
播放暂停，停止，继续播放

解码渲染之前用一个标识判断即可

seek

和音频播放类似，解码之前采用标识判断，当调用seek的时候设置标识，清除缓冲队列，
调用

1
avcodec_flush_buffers(&AVCodecContext);

进行seek,接着清空队列，并调用

1
avformat_seek_file(pFormatCtx, -1, INT64_MIN, rel, INT64_MAX, 0);

清空ffmpeg的缓存。

注意

  1. 这里有一个线程在使用AVFormatContext获取AvPacket,有一个线程在使用AVCodecContext在进行解码，需要为AVFormatContext和AVCodecContext添加锁。防止同步问题造成其他问题。
  2. 可能在seek之前，我们的数据已经读取完了存储在缓冲队列里面，这里seek清空缓冲队列，就会播放完毕，所以我们需要在读取不到数据的时候也加上seek标识判断。比如
1
2
3
4
5
6
7
8
9
10
11
12
13
//这里是读取数据完毕的时候
while(playstatus != NULL && !playstatus->exit){
    if(audio->queue->getQueueSize() > 0){
        av_usleep(1000 * 100);
        continue;
    } else{
        if(!playstatus->seek){
            av_usleep(1000 * 100);
            playstatus->exit = true;
        }
        break;
    }
}
release内存回收

这里需要特别注意的是线程退出的问题

单个线程退出

使用return 代替 pthread_exit();

多个线程退出

使用pthread_join(thread_t, NULL)，会阻塞当前线程，直到thread_t退出完。退出的时候需要理清楚线程的退出顺序。

发表于 | 分类于 |
字数统计: 1,216字 | 阅读时长 ≈ 7分钟
  1. 声明SoundTouch对象和内存变量,根据声道数和采样率初始化对象和内存
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
SoundTouch *soundTouch = NULL;
SAMPLETYPE *sampleBuffer = NULL;
//采样率
int sample_rate=44100;
//声道数
int channels =2;
//变调
float pitch= 1.0f;
//变数
float speed= 1.0f;
//采样位数 SoudTouch最低支持16bit，所以使用16bit的来播放
int bits= 16;
 //每秒理论PCM大小
int BUFF_SIZE =sample_rate * channels * bits/8;

   
sampleBuffer = static_cast<SAMPLETYPE *>(malloc(BUFF_SIZE));
soundTouch = new SoundTouch();
soundTouch->setSampleRate(sample_rate);
soundTouch->setChannels(channels);
soundTouch->setPitch(pitch);
soundTouch->setTempo(speed);
  1. 把PCM数据给SoundTouch处理
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
//采样个数，具体怎么获取看具体情况
int nb=0;
//示例1 ：文件读取
int size = fread();
nb = size/channels;
//示例2 ：ffmpeg解码
int nb = swr_convert();

//最大采样数
   
int maxSamples = BUFF_SIZE / channels; 

//处理数据
soundTouch->putSamples(sampleBuffer, nb);
//得到数据到sampleBuffer
int num = soundTouch->receiveSamples(sampleBuffer, maxSamples);
  1. 设置变速和变调
1
2
soundTouch->setPitch(1.0); //变调
soundTouch->setTempo(1.5);//变速
  1. SoudTouch选择处理数据是16bit还是32bit,在STTypes.h里面找到
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
#if !(SOUNDTOUCH_INTEGER_SAMPLES || SOUNDTOUCH_FLOAT_SAMPLES)
      
       /// Choose either 32bit floating point or 16bit integer sampletype
       /// by choosing one of the following defines, unless this selection 
       /// has already been done in some other file.
       ////
       /// Notes:
       /// - In Windows environment, choose the sample format with the
       ///   following defines.
       /// - In GNU environment, the floating point samples are used by 
       ///   default, but integer samples can be chosen by giving the 
       ///   following switch to the configure script:
       ///       ./configure --enable-integer-samples
       ///   However, if you still prefer to select the sample format here 
       ///   also in GNU environment, then please #undef the INTEGER_SAMPLE
       ///   and FLOAT_SAMPLE defines first as in comments above.
       //#define SOUNDTOUCH_INTEGER_SAMPLES     1    //< 16bit integer samples
       #define SOUNDTOUCH_FLOAT_SAMPLES       1    //< 32bit float samples
    
   #endif

根据你的类型注释选择对应的宏定义即可

  1. ffmpeg里面使用的时候需要注意的点：因为FFmpeg解码出来的PCM数据是8bit （uint8）的，而SoundTouch中最低
    是16bit（ 16bit integer samples），所以我们需要将8bit的数据转换成16bit
    后再给SoundTouch处理。

8bit->16bit处理方式:

1
2
3
4
5
6
7
8
9
10
11
SAMPLETYPE *sampleBuffer=NULL ;
uint8_t *out_buffer = NULL;

//....初始化等

//获取音频数据到out_buffer
int data_size = resampleAudio(reinterpret_cast<void **>(&out_buffer));
for(int i = 0; i < data_size / 2 + 1; i++)
{
	sampleBuffer[i] = (buffer[i * 2] | ((buffer[i * 2 + 1]) << 8));
}

  1. 官方示例，将一个文件变速变调转为另外一个文件
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
static void
_processFile(SoundTouch *pSoundTouch, const float pitch, const float tempo, const char *inFileName,
             const char *outFileName) {

    SoundTouch *pSoundTouch = new SoundTouch();
    
    //设置音调
    pSoundTouch->setPitch(pitch);
    //设置音速
    pSoundTouch->setTempo(tempo);
    
    int nSamples;//采样率
    int nChannels;//声道
    int buffSizeSamples;//每一次缓冲大小
    SAMPLETYPE sampleBuffer[BUFF_SIZE];//缓冲

    // open input file
    WavInFile inFile(inFileName);
    int sampleRate = inFile.getSampleRate();
    int bits = inFile.getNumBits();
    nChannels = inFile.getNumChannels();

    // create output file
    WavOutFile outFile(outFileName, sampleRate, bits, nChannels);

    pSoundTouch->setSampleRate(sampleRate);
    pSoundTouch->setChannels(nChannels);

    assert(nChannels > 0);
    buffSizeSamples = BUFF_SIZE / nChannels;

    // Process samples read from the input file
    while (inFile.eof() == 0) {
        int num;

        // Read a chunk of samples from the input file
        num = inFile.read(sampleBuffer, BUFF_SIZE);
        nSamples = num / nChannels;

        // Feed the samples into SoundTouch processor
        pSoundTouch->putSamples(sampleBuffer, nSamples);

        // Read ready samples from SoundTouch processor & write them output file.
        // NOTES:
        // - 'receiveSamples' doesn't necessarily return any samples at all
        //   during some rounds!
        // - On the other hand, during some round 'receiveSamples' may have more
        //   ready samples than would fit into 'sampleBuffer', and for this reason
        //   the 'receiveSamples' call is iterated for as many times as it
        //   outputs samples.
        do {
            nSamples = pSoundTouch->receiveSamples(sampleBuffer, buffSizeSamples);
            outFile.write(sampleBuffer, nSamples * nChannels);
        } while (nSamples != 0);
    }

    // Now the input file is processed, yet 'flush' few last samples that are
    // hiding in the SoundTouch's internal processing pipeline.
    pSoundTouch->flush();
    do {
        nSamples = pSoundTouch->receiveSamples(sampleBuffer, buffSizeSamples);
        outFile.write(sampleBuffer, nSamples * nChannels);
    } while (nSamples != 0);

    delete (pSoundTouch);
}
  1. ffmpeg示例
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

SoundTouch *soundTouch = NULL;
SAMPLETYPE *sampleBuffer = NULL;
//采样率
int sample_rate=44100;
//声道数
int channels =2;
//变调
float pitch= 1.0f;
//变数
float speed= 1.0f;
//采样位数 SoudTouch最低支持16bit，所以使用16bit的来播放
int bits= 16;
 //每秒理论PCM大小
int BUFF_SIZE =sample_rate * channels * bits/8;

   
sampleBuffer = static_cast<SAMPLETYPE *>(malloc(BUFF_SIZE));
soundTouch = new SoundTouch();
soundTouch->setSampleRate(sample_rate);
soundTouch->setChannels(channels);
soundTouch->setPitch(pitch);
soundTouch->setTempo(speed);


//获取SoundTouch处理的数据
int WlAudio::getSoundTouchData() {
    int maxSamples = BUFF_SIZE / channels;
    while (playstatus != NULL && !playstatus->exit) {
        out_buffer = NULL;
        if (finished) {
            finished = false;
            //获取pcm数据到out_buffer
            data_size = resampleAudio(reinterpret_cast<void **>(&out_buffer));
            if (data_size > 0) {
                for (int i = 0; i < data_size / 2 + 1; i++) {
                    //解码的数据是8bit的
                    //8bit->16bit
                    sampleBuffer[i] = (out_buffer[i * 2] | ((out_buffer[i * 2 + 1]) << 8));
                }
                //nb 表示采样个数  在resampleAudio里面解码的时候通过swr_convert返回值回去
                soundTouch->putSamples(sampleBuffer, nb);
                //获取处理后的数据  到sampleBuffer
                num = soundTouch->receiveSamples(sampleBuffer,maxSamples);
            } else {
                soundTouch->flush();
            }
        }
        if (num == 0) {
            finished = true;
            continue;
        } else {
            if (out_buffer == NULL) {
                num = soundTouch->receiveSamples(sampleBuffer, maxSamples);
                if (num == 0) {
                    finished = true;
                    continue;
                }
            }
            return num;
        }
    }
    return 0;
}

//OpenSLES播放数据

int buffersize = wlAudio->getSoundTouchData();
if (buffersize > 0) {
//两个8bit->一个16bit     转换为char*是为了都转换成字节来处理
 (*wlAudio->pcmBufferQueue)->Enqueue(wlAudio->pcmBufferQueue,
                                                (char *) wlAudio->sampleBuffer, buffersize * nChannels * 2 );
 }

发表于 | 分类于 |
字数统计: 270字 | 阅读时长 ≈ 2分钟

源码下载： https://gitlab.com/soundtouch/soundtouch, 源码下载之后里面有demo参考。
将下载的源码的include和source/SoundTouch目录的文件拷入

目录结构
目录结构

cmake

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
cmake_minimum_required(VERSION 3.4.1)

include_directories(src/main/cpp/soundtouch/include)
include_directories(src/main/cpp/soundtouch/SoundTouch)

add_library( # Sets the name of the library.
             native-lib

             # Sets the library as a shared library.
             SHARED

             # Provides a relative path to your source file(s).
              src/main/cpp/native-lib.cpp
              src/main/cpp/soundtouch/SoundTouch/AAFilter.cpp
              src/main/cpp/soundtouch/SoundTouch/FIFOSampleBuffer.cpp
              src/main/cpp/soundtouch/SoundTouch/FIRFilter.cpp
              src/main/cpp/soundtouch/SoundTouch/cpu_detect_x86.cpp
              src/main/cpp/soundtouch/SoundTouch/sse_optimized.cpp
              src/main/cpp/soundtouch/SoundTouch/RateTransposer.cpp
              src/main/cpp/soundtouch/SoundTouch/SoundTouch.cpp
              src/main/cpp/soundtouch/SoundTouch/InterpolateCubic.cpp
              src/main/cpp/soundtouch/SoundTouch/InterpolateLinear.cpp
              src/main/cpp/soundtouch/SoundTouch/InterpolateShannon.cpp
              src/main/cpp/soundtouch/SoundTouch/TDStretch.cpp
              src/main/cpp/soundtouch/SoundTouch/PeakFinder.cpp
              )


find_library( # Sets the name of the path variable.
              log-lib

              # Specifies the name of the NDK library that
              # you want CMake to locate.
              log )


target_link_libraries( # Specifies the target library.
                       native-lib

                       # Links the target library to the log library
                       # included in the NDK.
                       ${log-lib} )

配置信息参考demo

发表于 | 分类于 |
字数统计: 1,138字 | 阅读时长 ≈ 5分钟
每秒理论PCM大小
1
每秒理论PCM大小 = 采样率 * 声道数 * 位数/8

比如：

1
2
//44100hz 立体声 16bit
int s_time = 44100 * 2 * 16/8;

获取总时长
1
duration = pFormatCtx->duration / AV_TIME_BASE;
获取当前AVframe时间
1
2
3

AVRational time_base = pFormatCtx->streams[audio_index]->time_base
double now_time = frame->pts * av_q2d(time_base);
获取当前播放时间

因为每一个AVframe的pts不一定都有，所以就需要自己手维护一个当前时间的变量

1
2
公式：PCM实际数据大小 / 每秒理论PCM大小；
clock += buffersize / ((double)(sample_rate * 2 * 2));

伪代码如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
AVFormatContext *pFormatCtx = NULL;
//采样率
int sample_rate =0;
//当前总时长
int duration = 0;

AVRational time_base =NULL;
//当前AvFrame时间
int now_time=0;
//当前播放时长
int clock = 0;
//上次播放时长标识
int last_time=0;

uint8_t *buffer = NULL;
int data_size=0;


//获取到 pFormatCtx
...
for(int i = 0; i < pFormatCtx->nb_streams; i++)
    {
        if(pFormatCtx->streams[i]->codecpar->codec_type == AVMEDIA_TYPE_AUDIO)//得到音频流
        {
                ...
                sample_rate = pFormatCtx->streams[i]->codecpar->sample_rate;
                duration= pFormatCtx->duration / AV_TIME_BASE;
                time_base = pFormatCtx->streams[i]->time_base;
        }
}

//解码数据  
...
    
  int nb = swr_convert(
                swr_ctx,
                &buffer,
                avFrame->nb_samples,
                (const uint8_t **) avFrame->data,
                avFrame->nb_samples);

   int out_channels = av_get_channel_layout_nb_channels(AV_CH_LAYOUT_STEREO);
   data_size = nb * out_channels * av_get_bytes_per_sample(AV_SAMPLE_FMT_S16);

    now_time = avFrame->pts * av_q2d(time_base);
    if(now_time < clock){
       now_time = clock;
    }
    clock = now_time;


//解码数据之后数据封装在 buffer，播放的时候
...

if(data_size>0){
    //size/(采样率(44100hz) * 立体声*16bit/8) 
    clock += data_size / ((double)(sample_rate * 2 * 2));
    //设置一个回掉最小相差值
    if(clock - last_tiem >= 0.1) {
        last_tiem = clock;
        //回调应用层
       callJava->onCallTimeInfo(CHILD_THREAD,clock, duration);
    } 
}

解码播放流程思路

采用多线程，生产者消费者模型，AVPacket入队，然后AVPacket出队解码播放，播放采用OpenSLES

release内存回收

当我们release的时候，我们需要注意

  1. 为了确保线程完全退出，我们最好是sleep个几十毫秒，然后在释放相关内存,但是最好的是使用pthred_join来同步线程退出。
  2. 有可能初始化未准备完毕我们就调用release，这时候最好是在初始化准备和
    release加个线程锁。
  3. 初始化的时候有可能avformat_open_input打开网络链接，网络很卡，所以我们需要为pFormatCtx加入一个interrupt_callback来及时响应
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    int avformat_callback(void *ctx)
    {
        WlFFmpeg *fFmpeg = (WlFFmpeg *) ctx;
        if(fFmpeg->playstatus->exit)
        {
            return AVERROR_EOF;
        }
        return 0;
    }


    pFormatCtx->interrupt_callback.callback = avformat_callback;
    pFormatCtx->interrupt_callback.opaque = this;
暂停，继续,停止播放,播放完成

暂停播放，继续播放采用OpenSLES的相关api,播放完成则在播放完毕的时候回掉即可

seek功能

在seek的时候设置标志位并加锁，清空队列，标志位判断是否继续av_read_frame，seek完毕释放锁，还原标识位。即可重新读取最新数据

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

seek = true;
queue->clearAvpacket();

pthread_mutex_lock(&seek_mutex);
int64_t rel = secs * AV_TIME_BASE;
//重置内部解码器状态/刷新内部缓冲区
 avcodec_flush_buffers(avCodecContext);
//主要是这个函数
avformat_seek_file(pFormatCtx, -1, INT64_MIN, rel, INT64_MAX, 0);
pthread_mutex_unlock(&seek_mutex);
seek = false;

pthread_mutex_lock(&seek_mutex);
ret = av_read_frame(pFormatCtx, packet);
pthread_mutex_unlock(&seek_mutex);
音量,声道切换

采用OpenSLES的相关api

播放变速变调

OpenSL ES可以实现变速播放，但是再改变速度的同时也改变了音调，这
种体验是不好的。所以采用SoundTouch来实现，在播放的时候，对原始数据重新进行计算即可

计算pcm分贝大小
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

//char*是为了都转换成字节来处理 
int WlAudio::getPCMDB(char *pcmcata, size_t pcmsize) {
    int db = 0;
    short int pervalue = 0;
    double sum = 0;
    for (int i = 0; i < pcmsize; i += 2) {
        memcpy(&pervalue, pcmcata + i, 2);
        sum += abs(pervalue);
    }
    sum = sum / (pcmsize / 2);
    if (sum > 0) {
        db = (int) 20.0 * log10(sum);
    }
    return db;
}
性能优化
  1. 由于解码用到了while循环，而不加睡眠的while循环会使CPU使用率提高30%左右，
    因此我们需要为解码线程加上一定的睡眠时间来降低CPU使用率。
  2. 停止时回收创建的内存空间。
一个AVPacket对应多个AVFrame 比如.ape格式的

这种情况就需要在解码的时候，设置一个标识来判断不停的解析AVPacket,avcodec_send_packet(avCodecContext, avPacket)之后不停的avcodec_receive_frame，知道读取完毕在设置标识。

一个AVPacket对应多个AVFrame引发的seek问题

由于一个AVPacket里面有多个AVFrame，当seek时，FFmpeg解码器中还残留AVFrame，所以会导致seek后，不能立即播放当前音乐。
解决方案就是seek的时候调用

1
avcodec_flush_buffers(avCodecContext)

发表于 | 分类于 |
字数统计: 2,566字 | 阅读时长 ≈ 14分钟

  1. OpenSLES（Open Sound Library for Embedded Systems）无授权费、跨平台、针对嵌入式系统精心优化的硬件音频加速API。它为嵌入式移动多媒体设备上的本地应用程序开发者提供标准化, 高性能,低响应时间的音频功能实现方法，并实现软/硬件音频性能的直接跨平台部署，降低执行难度，促进高级音频市场的发展。

  2. 在Android里面ndk->platforms-> android-xx -> arch-xx ->usr->lib目录里面包含了ndk内置的so，可以看到支持了libOpenSLES.so。

  3. 在github上googlesamples/android-ndk可以看到ndk库的sample,里面native-audio目录就是OpenSLES得sample。

  1. OpenSLES播放主要步骤如下:
    1. 创建接口对象
    2. 设置混音器
    3. 创建播放器（录音器）
    4. 设置缓冲队列和回调函数
    5. 设置播放状态
    6. 启动回调函数
    7. 销毁

4.1 创建接口对象 

1
2
3
4
5
6
7
8
9

// 引擎接口
SLObjectItf engineObject = NULL;
SLEngineItf engineEngine = NULL;

// 创建引擎对象
 slCreateEngine(&engineObject,  0, NULL, 0, NULL, NULL);
 (*engineObject)->Realize(engineObject, SL_BOOLEAN_FALSE);
 (*engineObject)->GetInterface(engineObject, SL_IID_ENGINE, &engineEngine);

4.2 设置混音器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

//混音器
SLObjectItf outputMixObject = NULL;
SLEnvironmentalReverbItf outputMixEnvironmentalReverb = NULL;
SLEnvironmentalReverbSettings reverbSettings = SL_I3DL2_ENVIRONMENT_PRESET_STONECORRIDOR;


const SLInterfaceID mids[1] = {SL_IID_ENVIRONMENTALREVERB};
const SLboolean mreq[1] = {SL_BOOLEAN_FALSE};
(*engineEngine)->CreateOutputMix(engineEngine, &outputMixObject, 1, mids, mreq);
(void)result;
result = (*outputMixObject)->Realize(outputMixObject, SL_BOOLEAN_FALSE);
(void)result;
result = (*outputMixObject)->GetInterface(outputMixObject, SL_IID_ENVIRONMENTALREVERB, &outputMixEnvironmentalReverb);
if (SL_RESULT_SUCCESS == result) {
    result = (*outputMixEnvironmentalReverb)->SetEnvironmentalReverbProperties( outputMixEnvironmentalReverb, &reverbSettings);
    (void)result;
}

SLDataLocator_OutputMix outputMix = {SL_DATALOCATOR_OUTPUTMIX, outputMixObject};

4.3 创建播放器

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

 //pcm
 SLObjectItf pcmPlayerObject = NULL;
 SLPlayItf pcmPlayerPlay = NULL;
 SLVolumeItf pcmPlayerVolume = NULL;
 
 
 SLDataLocator_AndroidSimpleBufferQueue android_queue={SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE,2};
 SLDataFormat_PCM pcm={
             SL_DATAFORMAT_PCM,//播放pcm格式的数据
             2,//2个声道（立体声）
             SL_SAMPLINGRATE_44_1,//44100hz的频率
             SL_PCMSAMPLEFORMAT_FIXED_16,//位数 16位
             SL_PCMSAMPLEFORMAT_FIXED_16,//和位数一致就行
             SL_SPEAKER_FRONT_LEFT | SL_SPEAKER_FRONT_RIGHT,//立体声（前左前右）
             SL_BYTEORDER_LITTLEENDIAN//结束标志
  };
 
 SLDataSource slDataSource = {&android_queue, &pcm};
 SLDataSink audioSnk = {&outputMix, NULL};
// SL_IID_BUFFERQUEUE:缓冲  SL_IID_VOLUME:音量  SL_IID_PLAYBACKRATE:微调功能 防止卡顿 微调功能 SL_IID_MUTESOLO:声道切换
 const SLInterfaceID ids[4] = {SL_IID_BUFFERQUEUE, SL_IID_VOLUME, SL_IID_PLAYBACKRATE, SL_IID_MUTESOLO};
 const SLboolean req[4] = {SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE};
 
  result = (*engineEngine)->CreateAudioPlayer(engineEngine, &pcmPlayerObject, &slDataSource, &audioSnk, 3, ids, req);
     // 初始化播放器
 (*pcmPlayerObject)->Realize(pcmPlayerObject, SL_BOOLEAN_FALSE);
 
     //得到接口后调用  获取Player接口
 (*pcmPlayerObject)->GetInterface(pcmPlayerObject, SL_IID_PLAY, &pcmPlayerPlay);

4.4 设置缓冲队列和回调函数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

//缓冲器队列接口
SLAndroidSimpleBufferQueueItf pcmBufferQueue;
void *buffer;
uint8_t *out_buffer;

void getPcmData(void **pcm){
    while(!feof(pcmFile))  {
        int size = static_cast<int>(fread(out_buffer,1,44100 * 2 * 2,pcmFile));
        if(out_buffer == NULL)  {
            LOGI("%s  %d", "read end",size);
            break;
        } else{
            LOGI("%s  %d", "reading",size);
        }
        *pcm = out_buffer;
        break;
    }
}

void pcmBufferCallBack(SLAndroidSimpleBufferQueueItf bf, void * context){
    //assert(NULL == context);
    getPcmData(&buffer);
    // for streaming playback, replace this test by logic to find and fill the next buffer
    if (NULL != buffer) {
        SLresult result;
        // enqueue another buffer
        result = (*pcmBufferQueue)->Enqueue(pcmBufferQueue, buffer, 44100 * 2 * 2);
        // the most likely other result is SL_RESULT_BUFFER_INSUFFICIENT,
        // which for this code example would indicate a programming error
    }
}




// 创建缓冲区和回调函数
(*pcmPlayerObject)->GetInterface(pcmPlayerObject, SL_IID_BUFFERQUEUE, &pcmBufferQueue);

//缓冲接口回调
(*pcmBufferQueue)->RegisterCallback(pcmBufferQueue, pcmBufferCallBack, NULL);
//获取音量接口
(*pcmPlayerObject)->GetInterface(pcmPlayerObject, SL_IID_VOLUME, &pcmPlayerVolume);

4.5 设置播放状态

1
(*pcmPlayerPlay)->SetPlayState(pcmPlayerPlay, SL_PLAYSTATE_PLAYING);

4.6 启动回调函数

1
2
 // 主动调用回调函数开始工作
pcmBufferCallBack(pcmBufferQueue, NULL);

4.7 销毁

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22


 if (pcmPlayerObject != NULL) {
    (*pcmPlayerObject)->Destroy(pcmPlayerObject);
    pcmPlayerObject = NULL;
    pcmPlayerPlay = NULL;
    pcmBufferQueue = NULL;
    pcmPlayerVolume = NULL;
}


if (outputMixObject != NULL) {
    (*outputMixObject)->Destroy(outputMixObject);
    outputMixObject = NULL;
    outputMixEnvironmentalReverb = NULL;
}
  
 if (engineObject != NULL) {
    (*engineObject)->Destroy(engineObject);
    engineObject = NULL;
    engineEngine = NULL;
}

示例代码如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
#include <jni.h>
#include <string>


extern "C"
{
#include <SLES/OpenSLES.h>
#include <SLES/OpenSLES_Android.h>
}

#include <android/log.h>
#define LOGI(FORMAT,...) __android_log_print(ANDROID_LOG_INFO,"zzw",FORMAT,##__VA_ARGS__);
#define LOGE(FORMAT,...) __android_log_print(ANDROID_LOG_ERROR,"zzw",FORMAT,##__VA_ARGS__);

// 引擎接口
SLObjectItf engineObject = NULL;
SLEngineItf engineEngine = NULL;

//混音器
SLObjectItf outputMixObject = NULL;
SLEnvironmentalReverbItf outputMixEnvironmentalReverb = NULL;
SLEnvironmentalReverbSettings reverbSettings = SL_I3DL2_ENVIRONMENT_PRESET_STONECORRIDOR;


//pcm
SLObjectItf pcmPlayerObject = NULL;
SLPlayItf pcmPlayerPlay = NULL;
SLVolumeItf pcmPlayerVolume = NULL;

//缓冲器队列接口
SLAndroidSimpleBufferQueueItf pcmBufferQueue;

FILE *pcmFile;
void *buffer;

uint8_t *out_buffer;

void getPcmData(void **pcm)
{
    while(!feof(pcmFile))
    {
        int size = static_cast<int>(fread(out_buffer,1,44100 * 2 * 2,pcmFile));
        if(out_buffer == NULL)
        {
            LOGI("%s  %d", "read end",size);
            break;
        } else{
            LOGI("%s  %d", "reading",size);
        }
        *pcm = out_buffer;
        break;
    }
}

void pcmBufferCallBack(SLAndroidSimpleBufferQueueItf bf, void * context)
{
    //assert(NULL == context);
    getPcmData(&buffer);
    // for streaming playback, replace this test by logic to find and fill the next buffer
    if (NULL != buffer) {
        SLresult result;
        // enqueue another buffer
        result = (*pcmBufferQueue)->Enqueue(pcmBufferQueue, buffer, 44100 * 2 * 2);
        // the most likely other result is SL_RESULT_BUFFER_INSUFFICIENT,
        // which for this code example would indicate a programming error
    }
}

extern "C"
JNIEXPORT void JNICALL
Java_com_example_zzw_androidopenslaudio_MainActivity_palypcm(JNIEnv *env, jobject instance,
                                                               jstring url_) {
    const char *url = env->GetStringUTFChars(url_, 0);

    // TODO
    //读取pcm文件
    pcmFile = fopen(url, "r");
    if(pcmFile == NULL)
    {
        LOGE("%s", "fopen file error");
        return;
    }
    out_buffer = (uint8_t *) malloc(44100 * 2 * 2);


    SLresult result;
    //第一步------------------------------------------
    // 创建引擎对象
    slCreateEngine(&engineObject, 0, 0, 0, 0, 0);
    (*engineObject)->Realize(engineObject, SL_BOOLEAN_FALSE);
    (*engineObject)->GetInterface(engineObject, SL_IID_ENGINE, &engineEngine);


    //第二步-------------------------------------------
    // 创建混音器
    const SLInterfaceID mids[1] = {SL_IID_ENVIRONMENTALREVERB};
    const SLboolean mreq[1] = {SL_BOOLEAN_FALSE};
    result = (*engineEngine)->CreateOutputMix(engineEngine, &outputMixObject, 1, mids, mreq);
    (void)result;
    result = (*outputMixObject)->Realize(outputMixObject, SL_BOOLEAN_FALSE);
    (void)result;
    result = (*outputMixObject)->GetInterface(outputMixObject, SL_IID_ENVIRONMENTALREVERB, &outputMixEnvironmentalReverb);
    if (SL_RESULT_SUCCESS == result) {
        result = (*outputMixEnvironmentalReverb)->SetEnvironmentalReverbProperties(
                outputMixEnvironmentalReverb, &reverbSettings);
        (void)result;
    }
    SLDataLocator_OutputMix outputMix = {SL_DATALOCATOR_OUTPUTMIX, outputMixObject};

    // 第三步--------------------------------------------
    // 创建播放器
    SLDataLocator_AndroidSimpleBufferQueue android_queue={SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE,2};
    SLDataFormat_PCM pcm={
            SL_DATAFORMAT_PCM,//播放pcm格式的数据
            2,//2个声道（立体声）
            SL_SAMPLINGRATE_44_1,//44100hz的频率
            SL_PCMSAMPLEFORMAT_FIXED_16,//位数 16位
            SL_PCMSAMPLEFORMAT_FIXED_16,//和位数一致就行
            SL_SPEAKER_FRONT_LEFT | SL_SPEAKER_FRONT_RIGHT,//立体声（前左前右）
            SL_BYTEORDER_LITTLEENDIAN//结束标志
    };

    SLDataSource slDataSource = {&android_queue, &pcm};
    SLDataSink audioSnk = {&outputMix, NULL};
    
    // SL_IID_BUFFERQUEUE:缓冲  SL_IID_VOLUME:音量  SL_IID_PLAYBACKRATE:微调功能 防止卡顿 微调功能 SL_IID_MUTESOLO:声道切换
    const SLInterfaceID ids[4] = {SL_IID_BUFFERQUEUE, SL_IID_VOLUME, SL_IID_PLAYBACKRATE, SL_IID_MUTESOLO};
    const SLboolean req[4] = {SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE, SL_BOOLEAN_TRUE};

    result = (*engineEngine)->CreateAudioPlayer(engineEngine, &pcmPlayerObject, &slDataSource, &audioSnk, 3, ids, req);
    // 初始化播放器
    (*pcmPlayerObject)->Realize(pcmPlayerObject, SL_BOOLEAN_FALSE);

    //得到接口后调用  获取Player接口
    (*pcmPlayerObject)->GetInterface(pcmPlayerObject, SL_IID_PLAY, &pcmPlayerPlay);

    //第四步---------------------------------------
    // 创建缓冲区和回调函数
    (*pcmPlayerObject)->GetInterface(pcmPlayerObject, SL_IID_BUFFERQUEUE, &pcmBufferQueue);

    //缓冲接口回调
    (*pcmBufferQueue)->RegisterCallback(pcmBufferQueue, pcmBufferCallBack, NULL);
    //获取音量接口
    (*pcmPlayerObject)->GetInterface(pcmPlayerObject, SL_IID_VOLUME, &pcmPlayerVolume);

    //第五步----------------------------------------
    // 设置播放状态
    (*pcmPlayerPlay)->SetPlayState(pcmPlayerPlay, SL_PLAYSTATE_PLAYING);


    //第六步----------------------------------------
    // 主动调用回调函数开始工作
    pcmBufferCallBack(pcmBufferQueue, NULL);

    env->ReleaseStringUTFChars(url_, url);
}

  1. 暂停、继续、停止
    使用播放控制接口 SLPlayItf
1
2
3
4
5
6
7
8
9
10
11
12
//暂停
if (pcmPlayerPlay != NULL) {
    (*pcmPlayerPlay)->SetPlayState(pcmPlayerPlay, SL_PLAYSTATE_PAUSED);
}
//继续
if (pcmPlayerPlay != NULL) {
    (*pcmPlayerPlay)->SetPlayState(pcmPlayerPlay, SL_PLAYSTATE_PLAYING);
}
//停止
if (pcmPlayerPlay != NULL) {
    (*pcmPlayerPlay)->SetPlayState(pcmPlayerPlay, SL_PLAYSTATE_STOPPED);
}
  1. 音量控制
    使用音量控制接口 SLVolumeItf
1
2
3
4
//初始化
(*pcmPlayerObject)->GetInterface(pcmPlayerObject,SL_IID_VOLUME,&pcmPlayerVolume);
//设置音量
(*pcmPlayerVolume)->SetVolumeLevel(pcmPlayerVolume, (100 - percent) * -50);

可用示例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
void WlAudio::setVolume(int percent) {
    volumePercent = percent;
    if(pcmVolumePlay != NULL)
    {
        if(percent > 30)
        {
            (*pcmVolumePlay)->SetVolumeLevel(pcmVolumePlay, (100 - percent) * -20);
        }
        else if(percent > 25)
        {
            (*pcmVolumePlay)->SetVolumeLevel(pcmVolumePlay, (100 - percent) * -22);
        }
        else if(percent > 20)
        {
            (*pcmVolumePlay)->SetVolumeLevel(pcmVolumePlay, (100 - percent) * -25);
        }
        else if(percent > 15)
        {
            (*pcmVolumePlay)->SetVolumeLevel(pcmVolumePlay, (100 - percent) * -28);
        }
        else if(percent > 10)
        {
            (*pcmVolumePlay)->SetVolumeLevel(pcmVolumePlay, (100 - percent) * -30);
        }
        else if(percent > 5)
        {
            (*pcmVolumePlay)->SetVolumeLevel(pcmVolumePlay, (100 - percent) * -34);
        }
        else if(percent > 3)
        {
            (*pcmVolumePlay)->SetVolumeLevel(pcmVolumePlay, (100 - percent) * -37);
        }
        else if(percent > 0)
        {
            (*pcmVolumePlay)->SetVolumeLevel(pcmVolumePlay, (100 - percent) * -40);
        }
        else{
            (*pcmVolumePlay)->SetVolumeLevel(pcmVolumePlay, (100 - percent) * -100);
        }
    }
}

  1. 声道控制

采用声道控制接口SLMuteSoloItf接口

1
2
3
4
5
6
7
8
9
10
11
  SLMuteSoloItf  pcmMutePlay = NULL;

//初始化
  (*pcmPlayerObject)->GetInterface(pcmPlayerObject, SL_IID_MUTESOLO, &pcmMutePlay);
  
// 设置声道：
  (*pcmPlayPlayerMuteSolo)->SetChannelMute(
		pcmPlayPlayerMuteSolo, 
		1,  //0右声道1左声道
		false //声道是否开启
		);

有效示例:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
 SLMuteSoloItf  pcmMutePlay = NULL;
 //初始化
    ...
void WlAudio::setMute(int mute) {
    this->mute = mute;
    if(pcmMutePlay != NULL)
    {
        if(mute == 0)//right
        {
            (*pcmMutePlay)->SetChannelMute(pcmMutePlay, 1, false);
            (*pcmMutePlay)->SetChannelMute(pcmMutePlay, 0, true);
        }
        else if(mute == 1)//left
        {
            (*pcmMutePlay)->SetChannelMute(pcmMutePlay, 1, true);
            (*pcmMutePlay)->SetChannelMute(pcmMutePlay, 0, false);
        }
        else if(mute == 2)//center
        {
            (*pcmMutePlay)->SetChannelMute(pcmMutePlay, 1, false);
            (*pcmMutePlay)->SetChannelMute(pcmMutePlay, 0, false);
        }
    }
}

  1. 录音

有效示例:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136

#include <jni.h>
#include <string>
#include "AndroidLog.h"
#include "RecordBuffer.h"


#include <SLES/OpenSLES.h>
#include <SLES/OpenSLES_Android.h>

bool finish = false;
FILE *recodeFile = NULL;


const static int RECORDER_BUFFER_SIZE = 4096;

SLObjectItf engineObject = NULL;
SLEngineItf engineItf = NULL;

SLObjectItf recordObj = NULL;
SLRecordItf recordItf = NULL;
SLAndroidSimpleBufferQueueItf recorderBufferQueue = NULL;

RecordBuffer *recordBuffer = NULL;

// this callback handler is called every time a buffer finishes recording
void bqRecorderCallback(SLAndroidSimpleBufferQueueItf bq, void *context) {
    fwrite(recordBuffer->getNowBuffer(), 1, RECORDER_BUFFER_SIZE * sizeof(short), recodeFile);
    if (finish) {
        LOGE("录制完成");
        //设置停止
        (*recordItf)->SetRecordState(recordItf, SL_RECORDSTATE_STOPPED);

        fclose(recodeFile);

        //释放资源
        (*recordObj)->Destroy(recordObj);
        recordObj = NULL;
        recordItf = NULL;
        (*engineObject)->Destroy(engineObject);
        engineObject = NULL;
        engineItf = NULL;
        delete (recordBuffer);

    } else {
        LOGE("正在录制");
        // 入队
        (*recorderBufferQueue)->Enqueue(recorderBufferQueue, recordBuffer->getRecordBuffer(),
                                        RECORDER_BUFFER_SIZE * sizeof(short));
    }

}

extern "C"
JNIEXPORT void JNICALL
Java_com_zzw_openslesrecoder_MainActivity_startRecord(JNIEnv *env, jobject instance,
                                                      jstring path_) {

    const char *path = env->GetStringUTFChars(path_, 0);
    finish = false;

    recodeFile = fopen(path, "w+");

    //1. 创建引擎对象
    slCreateEngine(&engineObject, 0, NULL, 0, NULL, NULL);

    //2. 实现引擎对象
    (*engineObject)->Realize(engineObject, SL_BOOLEAN_FALSE);

    //3. 获取引擎接口
    (*engineObject)->GetInterface(engineObject, SL_IID_ENGINE, &engineItf);



    // configure audio source
    SLDataLocator_IODevice loc_dev = {SL_DATALOCATOR_IODEVICE, SL_IODEVICE_AUDIOINPUT,
                                      SL_DEFAULTDEVICEID_AUDIOINPUT, NULL};
    SLDataSource audioSrc = {&loc_dev, NULL};

    // configure audio sink
    SLDataLocator_AndroidSimpleBufferQueue loc_bq = {SL_DATALOCATOR_ANDROIDSIMPLEBUFFERQUEUE, 2};
    SLDataFormat_PCM format_pcm = {SL_DATAFORMAT_PCM, //PCM格式
                                   2,//立体声
                                   SL_SAMPLINGRATE_44_1,//44100HZ
                                   SL_PCMSAMPLEFORMAT_FIXED_16,//
                                   SL_PCMSAMPLEFORMAT_FIXED_16,
                                   SL_SPEAKER_FRONT_LEFT | SL_SPEAKER_FRONT_RIGHT, //左右声道
                                   SL_BYTEORDER_LITTLEENDIAN};//小尾端
    SLDataSink audioSnk = {&loc_bq, &format_pcm};


    // (requires the RECORD_AUDIO permission)
    const SLInterfaceID id[1] = {SL_IID_ANDROIDSIMPLEBUFFERQUEUE};
    const SLboolean req[1] = {SL_BOOLEAN_TRUE};

    //4. 配置获取录音的引擎对象
    (*engineItf)->CreateAudioRecorder(engineItf, &recordObj, &audioSrc,
                                      &audioSnk, 1, id, req);

    //5. 实现录音的引擎对象
    // realize the audio recorder
    (*recordObj)->Realize(recordObj, SL_BOOLEAN_FALSE);

    //6. 获取录音的引擎接口
    //get the record interface
    (*recordObj)->GetInterface(recordObj, SL_IID_RECORD, &recordItf);


    //7. 获取缓冲队列接口
    //get the buffer queue interface
    (*recordObj)->GetInterface(recordObj, SL_IID_ANDROIDSIMPLEBUFFERQUEUE,
                               &recorderBufferQueue);

    //8. 设置录音回掉
    (*recorderBufferQueue)->RegisterCallback(recorderBufferQueue, bqRecorderCallback,
                                             NULL);

    recordBuffer = new RecordBuffer(RECORDER_BUFFER_SIZE);
    //9. 入队
    (*recorderBufferQueue)->Enqueue(recorderBufferQueue, recordBuffer->getRecordBuffer(),
                                    RECORDER_BUFFER_SIZE * sizeof(short));

    //10. 设置状态开启录音
    (*recordItf)->SetRecordState(recordItf, SL_RECORDSTATE_RECORDING);
    env->ReleaseStringUTFChars(path_, path);
}




extern "C"
JNIEXPORT void JNICALL
Java_com_zzw_openslesrecoder_MainActivity_stopRecord(JNIEnv *env, jobject instance) {

    finish = true;
}

RecordBuffer.cpp:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

#include "RecordBuffer.h"



RecordBuffer::RecordBuffer(int bufferSize) {
    buffer = new short *[2];
    for (int i = 0; i < 2; i++) {
        buffer[i] = new short[bufferSize];
    }
}

short *RecordBuffer::getRecordBuffer() {
    index++;
    if (index > 1) {
        index = 0;
    }
    return buffer[index];
}

RecordBuffer::~RecordBuffer() {
    for (int i = 0; i < 2; i++) {
        delete buffer[i];
    }
    delete buffer;
}

short *RecordBuffer::getNowBuffer() {
    return buffer[index];
}

RecordBuffer.h

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
class RecordBuffer {

public:
    short **buffer;
    int index = 0;

public:
    RecordBuffer(int bufferSize);
    ~RecordBuffer();

    short *getRecordBuffer();

    short * getNowBuffer();


};

发表于 | 分类于 |
字数统计: 469字 | 阅读时长 ≈ 2分钟

生产者消费者模型主要有以下函数和对象

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
//线程锁对象
pthread_mutex_t mutex;

//用于初始化pthread_mutex_t锁对象 
pthread_mutex_init(&mutex, NULL);

//用于销毁pthread_mutex_t锁对象
pthread_mutex_destroy(&mutex)

//线程条件对象
pthread_cond_t cond;
//用于初始化pthread_cond_t线程条件对象 
pthread_cond_init(&cond, NULL);
//用于销毁pthread_cond_t线程条件对象 
pthread_cond_destroy(&cond);

//用于上锁mutex,本线程上锁后的其他变量是不能
被别的线程操作
pthread_mutex_lock(&mutex);

//用于解锁mutex，解锁后的其他变量可以被其他线程操作
pthread_mutex_unlock(&mutex);

//用于发出条件信号
 pthread_cond_signal(&cond);

//用于线程阻塞等待，这个函数会解锁，直到pthread_cond_signal发出条件信号后才执行退出线程阻塞执行后面的操作
才执行退出线程阻塞执行后面的操作
pthread_cond_wait(&cond, &mutex);

示例代码如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70


#include "pthread.h"
#include "queue"
#include "unistd.h"

pthread_t produc;
pthread_t custom;
pthread_mutex_t mutex;
pthread_cond_t cond;

std::queue<int> queue;

void *producCallback(void *data)
{

    while (1)
    {
        pthread_mutex_lock(&mutex);

        queue.push(1);
        LOGD("生产者生产一个产品，通知消费者消费， 产品数量为 %d", queue.size());
        pthread_cond_signal(&cond);
        pthread_mutex_unlock(&mutex);
        sleep(5);
    }
    pthread_exit(&produc);

}

void *customCallback(void *data)
{
    while (1)
    {
        pthread_mutex_lock(&mutex);

        if(queue.size() > 0)
        {
            queue.pop();
            LOGD("消费者消费产品，产品数量还剩余 %d ", queue.size());
        } else{
            LOGD("没有产品可以消费， 等待中...");
            pthread_cond_wait(&cond, &mutex);
        }
        pthread_mutex_unlock(&mutex);
        usleep(500 * 1000);
    }
    pthread_exit(&custom);
    pthread_mutex_destroy(&mutex);
    pthread_cond_destroy(&cond);

}


extern "C"
JNIEXPORT void JNICALL
Java_com_zzw_jnithread_ThreadDemo_mutexThread(JNIEnv *env, jobject instance) {


    for(int i = 0; i < 10; i++)
    {
        queue.push(1);
    }
    pthread_mutex_init(&mutex, NULL);
    pthread_cond_init(&cond, NULL);

    pthread_create(&produc, NULL, producCallback, NULL);
    pthread_create(&custom, NULL, customCallback, NULL);

}

发表于 | 分类于 |
字数统计: 1,156字 | 阅读时长 ≈ 6分钟

libswscale里面实现了各种图像像素格式的转换。

主要函数如下:

1
2
3
4
5
6
//使用参数初始化SwsContext结构体
sws_getContext()
//转换一帧图像
sws_scale()
//释放SwsContext结构体
sws_freeContext()

初始化函数:

1
2
3
4
5
6
//为SwsContext结构体分配内存
sws_alloc_context()
//设置SwsContext结构体的值
av_opt_set()/av_opt_set_XXX()
//初始化SwsContext结构体。
sws_init_context()

这种复杂的方法可以配置一些sws_getContext()配置不了的参数。比如说设置图像的YUV像素的取值范围是JPEG标准（Y、U、V取值范围都是0-255）还是MPEG标准（Y取值范围是16-235，U、V的取值范围是16-240）

获取像素格式信息:

1
2
3
4
//可以获得指定像素格式的AVPixFmtDescriptor结构体
av_pix_fmt_desc_get()
//通过AVPixFmtDescriptor获取值
av_get_bits_per_pixel() //获取比特数（bpp）

图像拉伸:

1
SWS_BICUBIC性能比较好；SWS_FAST_BILINEAR在性能和速度之间有一个比好好的平衡。

示例将通过yuv文件生成rgb文件，代码如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
#include "stdafx.h"


extern "C" {
	#include <libswscale/swscale.h>
	#include <libavutil/opt.h>
	#include <libavutil/imgutils.h>
}


void p(const char * msg, int d = -1123) {
	if (d == -1123) {
		printf_s("%s\n", msg);
	}
	else {
		printf_s("%s  %d \n", msg, d);
	}
}


int yuv2Rgb(FILE *yuvFile,FILE * rgbFile) {

	SwsContext *img_convert_ctx = NULL;

	const AVPixelFormat srcPixelFormat = AV_PIX_FMT_YUV420P;
	const AVPixelFormat dstPixelFormat = AV_PIX_FMT_RGB24;

	const int srcW = 1080;
	const int srcH= 720;

	const int dstW = 540;
	const int dstH = 360;

	uint8_t *src_data[4];
	int src_linesize[4];

	uint8_t *dst_data[4];
	int dst_linesize[4];


	//1. 获得指定像素格式的AVPixFmtDescriptor结构体
	const AVPixFmtDescriptor * pSrcPixFmtDes = av_pix_fmt_desc_get(srcPixelFormat);
	//2. 获得指定像素格式每个像素占用的比特数bpp（Bit Per Pixel） 
	int srcBpp=  av_get_bits_per_pixel(pSrcPixFmtDes);

	const AVPixFmtDescriptor * pDstPixFmtDes = av_pix_fmt_desc_get(dstPixelFormat);
	int dstBpp = av_get_bits_per_pixel(pDstPixFmtDes);


	

	//3. 根据宽高，像素格式分配buffer大小
	if (av_image_alloc(src_data, src_linesize, srcW, srcH, srcPixelFormat, 1) < 0) {
		p("Could not allocate source image");
		return -1;
	}

	if (av_image_alloc(dst_data, dst_linesize, dstW, dstH, dstPixelFormat, 1) < 0) {
		p("Could not allocate source image");
		return -1;
	}

	//4. 为SwsContext结构体分配内存。
	img_convert_ctx = sws_alloc_context();
	//Show AVOption
	av_opt_show2(img_convert_ctx, stdout, AV_OPT_FLAG_VIDEO_PARAM, 0);




	//图像拉伸   SWS_BICUBIC性能比较好；SWS_FAST_BILINEAR在性能和速度之间有一个比好好的平衡。
	//const int rescale_method = SWS_BICUBIC;
	//5. 设置值
	av_opt_set_int(img_convert_ctx, "sws_flags", SWS_BICUBIC | SWS_PRINT_INFO, 0);
	av_opt_set_int(img_convert_ctx, "srcw", srcW, 0);
	av_opt_set_int(img_convert_ctx, "srch", srcH, 0);
	av_opt_set_int(img_convert_ctx, "src_format", srcPixelFormat, 0);
	//'0' for MPEG (Y:0-235);'1' for JPEG (Y:0-255)
	av_opt_set_int(img_convert_ctx, "src_range", 1, 0);
	av_opt_set_int(img_convert_ctx, "dstw", dstW, 0);
	av_opt_set_int(img_convert_ctx, "dsth", dstH, 0);
	av_opt_set_int(img_convert_ctx, "dst_format", dstPixelFormat, 0);
	av_opt_set_int(img_convert_ctx, "dst_range", 1, 0);
	sws_init_context(img_convert_ctx, NULL, NULL);//对SwsContext中的各种变量进行赋值

	
	uint8_t *temp_buffer = (uint8_t *)malloc(srcW *srcH *srcBpp / 8);

	int frame_idx = 0;
	while (1){
		if (fread(temp_buffer, 1, srcW*srcH*srcBpp / 8, yuvFile) != srcW * srcH*srcBpp / 8) {
			break;
		}

		switch (srcPixelFormat) {
			case AV_PIX_FMT_GRAY8: {
				memcpy(src_data[0], temp_buffer, srcW*srcH);
				break;
			}
			case AV_PIX_FMT_YUV420P: {
				memcpy(src_data[0], temp_buffer, srcW*srcH);                    //Y
				memcpy(src_data[1], temp_buffer + srcW * srcH, srcW*srcH / 4);      //U
				memcpy(src_data[2], temp_buffer + srcW * srcH * 5 / 4, srcW*srcH / 4);  //V
				break;
			}
			case AV_PIX_FMT_YUV422P: {
				memcpy(src_data[0], temp_buffer, srcW*srcH);                    //Y
				memcpy(src_data[1], temp_buffer + srcW * srcH, srcW*srcH / 2);      //U
				memcpy(src_data[2], temp_buffer + srcW * srcH * 3 / 2, srcW*srcH / 2);  //V
				break;
			}
			case AV_PIX_FMT_YUV444P: {
				memcpy(src_data[0], temp_buffer, srcW*srcH);                    //Y
				memcpy(src_data[1], temp_buffer + srcW * srcH, srcW*srcH);        //U
				memcpy(src_data[2], temp_buffer + srcW * srcH * 2, srcW*srcH);      //V
				break;
			}
			case AV_PIX_FMT_YUYV422: {
				memcpy(src_data[0], temp_buffer, srcW*srcH * 2);                  //Packed
				break;
			}
			case AV_PIX_FMT_RGB24: {
				memcpy(src_data[0], temp_buffer, srcW*srcH * 3);                  //Packed
				break;
			}
			default: {
				printf("Not Support Input Pixel Format.\n");
				break;
			}
		}

		sws_scale(img_convert_ctx, src_data, src_linesize, 0, srcH, dst_data, dst_linesize);//转换像素
		printf("Finish process frame %5d\n", frame_idx);
		frame_idx++;

		switch (dstPixelFormat) {
			case AV_PIX_FMT_GRAY8: {
				fwrite(dst_data[0], 1, dstW*dstH, rgbFile);
				break;
			}
			case AV_PIX_FMT_YUV420P: {
				fwrite(dst_data[0], 1, dstW*dstH, rgbFile);                 //Y
				fwrite(dst_data[1], 1, dstW*dstH / 4, rgbFile);               //U
				fwrite(dst_data[2], 1, dstW*dstH / 4, rgbFile);               //V
				break;
			}
			case AV_PIX_FMT_YUV422P: {
				fwrite(dst_data[0], 1, dstW*dstH, rgbFile);                 //Y
				fwrite(dst_data[1], 1, dstW*dstH / 2, rgbFile);               //U
				fwrite(dst_data[2], 1, dstW*dstH / 2, rgbFile);               //V
				break;
			}
			case AV_PIX_FMT_YUV444P: {
				fwrite(dst_data[0], 1, dstW*dstH, rgbFile);                 //Y
				fwrite(dst_data[1], 1, dstW*dstH, rgbFile);                 //U
				fwrite(dst_data[2], 1, dstW*dstH, rgbFile);                 //V
				break;
			}
			case AV_PIX_FMT_YUYV422: {
				fwrite(dst_data[0], 1, dstW*dstH * 2, rgbFile);               //Packed
				break;
			}
			case AV_PIX_FMT_RGB24: {
				fwrite(dst_data[0], 1, dstW*dstH * 3, rgbFile);               //Packed
				break;
			}
			default: {
				p("Not Support Output Pixel Format.\n");
				break;
			}
		}
	}

	sws_freeContext(img_convert_ctx);
	free(temp_buffer);
	av_freep(&src_data[0]);
	av_freep(&dst_data[0]);

	return 0;


}


int main() {
	FILE* inFile;
	FILE* outFile;
	fopen_s(&inFile,"F:/视频资源/gxsp.yuv", "rb");
	fopen_s(&outFile, "F:/视频资源/gxsp.rgb", "wb");
	yuv2Rgb(inFile,outFile);

	fclose(inFile);
	fclose(outFile);

	getchar();

	return 0;
}

参考链接:
libswscale实现YUV转RGB

相关格式转换:
视音频数据处理入门：RGB、YUV像素数据处理

发表于 | 分类于 |
字数统计: 1,301字 | 阅读时长 ≈ 7分钟

格式转换直接将视音频压缩码流从一种封装格式文件中获取出来然后打包成另外一种封装格式的文件。因为不需要进行视音频的编码和解码，所以不会有视音频的压缩损伤。

主要步骤如下:

  1. 注册初始化
1
av_register_all();
  1. 打开输入文件，获取视频相关信息
1
2
avformat_open_input()
avformat_find_stream_info()
  1. 初始化输出文件的AVFormatContext并copy输入文件AVFormatContext的相关设置，打开输出文件
1
2
3
4
5
6
7
//为输出格式分配AVFormatContext
avformat_alloc_output_context2()
//【关键步骤】 copy输入文件的设置到输出文件
avcodec_copy_context()

//打开输出文件
avio_open()
  1. 写入数据到文件
1
2
3
4
5
6
7
8
9
//将文件流头部数据写入文件
avformat_write_header()
//读取帧
av_read_frame()

//数据包写入文件
av_interleaved_write_frame()
//将文件流尾部数据写入文件并释放文件资源
av_write_trailer()
  1. 回收

具体代码和步骤如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149

#include <stdio.h>
#include "stdafx.h"


extern "C"
{
#include <libavformat/avformat.h>
};


void log_ss(const char * msg, int d = -1123) {
	if (d == -1123) {
		printf_s("%s\n", msg);
	}
	else {
		printf_s("%s  %d \n", msg, d);
	}
}

int main()
{
    AVOutputFormat *ofmt = NULL;
    //Input AVFormatContext and Output AVFormatContext
    AVFormatContext *ifmt_ctx = NULL, *ofmt_ctx = NULL;
    AVPacket pkt;
    int ret, i;
    int frame_index=0;
	const char * in_filename  = "F:/视频资源/gxsp.mp4";//Input file URL
	const char *  out_filename = "F:/视频资源/gxsp.avi";//Output file URL


	//1. 注册初始化
	//H.264 bitstream malformed, no startcode found, use the video bitstream filte错误解决方法   第一步
	AVBitStreamFilterContext *filter = av_bitstream_filter_init("h264_mp4toannexb");
	if (NULL == filter)
	{
		log_ss("filter init fail");
	}
	//av_register_bitstream_filter(filter->filter);
    av_register_all();
	//2. 打开媒体文件与AVFormatContext关联
    //Input
    if ((ret = avformat_open_input(&ifmt_ctx, in_filename, 0, 0)) < 0) {
		log_ss( "Could not open input file.");
        goto end;
    }
	//3. 获取视频流信息
    if ((ret = avformat_find_stream_info(ifmt_ctx, 0)) < 0) {//
		log_ss( "Failed to retrieve input stream information");
        goto end;
    }
	log_ss("--------------- In File Information ----------------");
    av_dump_format(ifmt_ctx, 0, in_filename, 0);
	log_ss("--------------- In File Information ----------------");
    //4. 初始化输出视频码流的AVFormatContext，与输出文件相关联
    avformat_alloc_output_context2(&ofmt_ctx, NULL, NULL, out_filename);
    if (!ofmt_ctx) {
		log_ss( "Could not create output context");
        ret = AVERROR_UNKNOWN;
        goto end;
    }

	//获取AVOutputFormat
    ofmt = ofmt_ctx->oformat;
    for (i = 0; i < ifmt_ctx->nb_streams; i++) {
        //Create output AVStream according to input AVStream
        AVStream *in_stream = ifmt_ctx->streams[i];
        AVStream *out_stream = avformat_new_stream(ofmt_ctx, in_stream->codec->codec);// 初始化AVStream 
        if (!out_stream) {
			log_ss( "Failed allocating output stream\n");
            ret = AVERROR_UNKNOWN;
            goto end;
        }
		//5.【关键步骤】 copy输入文件的设置到输出文件
		  if (avcodec_copy_context(out_stream->codec, in_stream->codec) < 0) {
				log_ss( "Failed to copy context from input to output stream codec context\n");
			  goto end;
		   }
        out_stream->codec->codec_tag = 0;
        if (ofmt_ctx->oformat->flags & AVFMT_GLOBALHEADER)
            out_stream->codec->flags |= CODEC_FLAG_GLOBAL_HEADER;
    }
	log_ss("--------------- Out File Information ----------------");
    av_dump_format(ofmt_ctx, 0, out_filename, 1);
	log_ss("--------------- Out File Information ----------------");
	//6. 打开输出文件。
    if (!(ofmt->flags & AVFMT_NOFILE)) {
        ret = avio_open(&ofmt_ctx->pb, out_filename, AVIO_FLAG_WRITE);
        if (ret < 0) {
			log_ss( "Could not open output file");
			log_ss(  out_filename);
            goto end;
        }
    }
    //7. 写入头部到文件
    if (avformat_write_header(ofmt_ctx, NULL) < 0) {
		log_ss( "Error occurred when opening output file\n");
        goto end;
    }
	
	//8. 写入数据到文件
    while (1) {
        AVStream *in_stream, *out_stream;
        //Get an AVPacket
        ret = av_read_frame(ifmt_ctx, &pkt);
        if (ret < 0)
            break;
        in_stream  = ifmt_ctx->streams[pkt.stream_index];
        out_stream = ofmt_ctx->streams[pkt.stream_index];

		//H.264 bitstream malformed, no startcode found, use the video bitstream filte错误解决方法   第二步
		if (pkt.stream_index == 0) {
			AVPacket fpkt = pkt;
			int a = av_bitstream_filter_filter(filter,
				out_stream->codec, NULL, &fpkt.data, &fpkt.size,
				pkt.data, pkt.size, pkt.flags & AV_PKT_FLAG_KEY);
			pkt.data = fpkt.data;
			pkt.size = fpkt.size;
		}

        //Convert PTS/DTS
        pkt.pts = av_rescale_q_rnd(pkt.pts, in_stream->time_base, out_stream->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
        pkt.dts = av_rescale_q_rnd(pkt.dts, in_stream->time_base, out_stream->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));
        pkt.duration = av_rescale_q(pkt.duration, in_stream->time_base, out_stream->time_base);
        pkt.pos = -1;
		//将AVPacket（存储视频压缩码流数据）写入文件   av_interleaved_write_frame将对 packet 进行缓存和 pts 检查，av_write_frame没有
        if (av_interleaved_write_frame(ofmt_ctx, &pkt) < 0) {
			log_ss( "Error muxing packets");
            break;
        }
        log_ss("Write frames to output file:",frame_index);
		av_packet_unref(&pkt);
        frame_index++;
    }

	//9. 写入文件尾部 
    //Write file trailer
    av_write_trailer(ofmt_ctx);
end:
    avformat_close_input(&ifmt_ctx);
    /* close output */
    if (ofmt_ctx && !(ofmt->flags & AVFMT_NOFILE))
        avio_close(ofmt_ctx->pb);
    avformat_free_context(ofmt_ctx);

	getchar();
    return 0;
}

遇到的错误:

  1. 错误C4996 ‘xxxx’: 被申明已否决
    原因: 因为api已经过时
    解决: 建议换用新的api，或者在属性->c/c++->常规里面关闭SDL检查

  2. H.264 bitstream malformed, no startcode found, use the video bitstream filte
    原因：主要是因为使用了mp4中的h264编码，而h264有两种封装：
    一种是annexb模式，传统模式，有startcode，SPS和PPS是在ES中；另一种是mp4模式，一般mp4、mkv、avi会没有startcode，SPS和PPS以及其它信息被封装在container中，每一个frame前面是这个frame的长度，很多解码器只支持annexb这种模式，因此需要将mp4做转换；在ffmpeg中用h264_mp4toannexb_filter可以做转换；所以需要使用-bsf h264_mp4toannexb来进行转换；
    解决：
    (1) 注册过滤器

1
2
3
4
5
6
AVBitStreamFilterContext *filter = av_bitstream_filter_init("h264_mp4toannexb");
    if (NULL == filter)
    {
        cout<<"filter init fail"<<endl;
    }
    av_register_bitstream_filter(filter->filter);

(2) 在获取第一帧的时候进行处理

1
2
3
4
5
6
7
8
if (pkt.stream_index == 0) {
			AVPacket fpkt = pkt;
			int a = av_bitstream_filter_filter(filter,
				out_stream->codec, NULL, &fpkt.data, &fpkt.size,
				pkt.data, pkt.size, pkt.flags & AV_PKT_FLAG_KEY);
			pkt.data = fpkt.data;
			pkt.size = fpkt.size;
		}

参考:
ffmpeg实战教程（四）格式转换如MP4转MKV等
H.264 bitstream malformed, no startcode found, use the video bitstream filte错误解决方法