How to make librtmp compile on Mac OS X - macos

Question

I am trying to build rtmpdump and librtmp on mac, however I cannot for the life of me work out why it isn't working. I am using my own built openssl:
$ openssl version
OpenSSL 1.1.0-pre3 (alpha) 15 Feb 2016
I have set the flags correctly (I think), but yet when I try and compile I get this:
$ make INC=-I/usr/local/include
gcc -Wall -I/usr/local/include -DRTMPDUMP_VERSION=\"v2.4\" -DUSE_OPENSSL -O2 -fPIC -c -o rtmp.o rtmp.c
In file included from rtmp.c:152:
./handshake.h:120:12: error: variable has incomplete type 'HMAC_CTX' (aka 'struct hmac_ctx_st')
HMAC_CTX ctx;
^
/usr/local/include/openssl/ossl_typ.h:145:16: note: forward declaration of 'struct hmac_ctx_st'
typedef struct hmac_ctx_st HMAC_CTX;
^
In file included from rtmp.c:152:
./handshake.h:125:3: warning: implicit declaration of function 'HMAC_CTX_init' is invalid in C99 [-Wimplicit-function-declaration]
HMAC_setup(ctx, secretKey, 128);
^
./handshake.h:72:35: note: expanded from macro 'HMAC_setup'
#define HMAC_setup(ctx, key, len) HMAC_CTX_init(&ctx); HMAC_Init_ex(&ctx, key, len, EVP_sha256(), 0)
^
./handshake.h:127:3: warning: implicit declaration of function 'HMAC_CTX_cleanup' is invalid in C99 [-Wimplicit-function-declaration]
HMAC_finish(ctx, digest, digestLen);
^
./handshake.h:74:67: note: expanded from macro 'HMAC_finish'
#define HMAC_finish(ctx, dig, dlen) HMAC_Final(&ctx, dig, &dlen); HMAC_CTX_cleanup(&ctx)
^
./handshake.h:269:12: error: variable has incomplete type 'HMAC_CTX' (aka 'struct hmac_ctx_st')
HMAC_CTX ctx;
^
/usr/local/include/openssl/ossl_typ.h:145:16: note: forward declaration of 'struct hmac_ctx_st'
typedef struct hmac_ctx_st HMAC_CTX;
^
rtmp.c:192:13: warning: comparison of constant 18446744073709551597 with expression of type 'uint32_t' (aka 'unsigned int') is always false [-Wtautological-constant-out-of-range-compare]
if (nSize > SIZE_MAX - RTMP_MAX_HEADER_SIZE)
~~~~~ ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
3 warnings and 2 errors generated.
make[1]: *** [rtmp.o] Error 1
make: *** [librtmp/librtmp.a] Error 2
It looks like it might be a compatibility issue with the version of OpenSSL I'm using. Does anyone know how to fix this kind of issue, or any changes to OpenSSL that might be responsible?

Answer 1

OpenSSL 1.1.0-pre3 (alpha) 15 Feb 2016...
Does anyone know how to fix this kind of issue, or any changes to OpenSSL that might be responsible?
Broadly speaking, there appears to be two high level changes in OpenSSL 1.1.x that caused most of the problems. First was the changes to the build system. Second was the firmer posture with respect to private APIs. I think you're seeing an artifact of the second, the private APIs.
Generally speaking, 1.1.0-pre3 and 1.1.0-pre4 have a number of problems and mostly can't achieve a successful make test. The issues include configuration and build issues for 32-bit OS X PowerMac, 64-bit OS X PowerMac, 32-bit OS X, 64-bit OS X, 32-bit ARM, 64-bit ARM, OpenBSD 5.7, NetBSD 7.0, Cygwin, CentOS 5, CentOS 7 and i686. So many so that I've filed about 25 bugs in preparation for the upcoming Beta-1 scheduled to arrive Monday, 14 MAR 2016.
You should probably avoid 1.1.0-pre-N at the moment because its so unstable. You should be OK under x86_64 Linux, but anything else is a crap shoot at the moment. You should be able to pick it up again around Beta-2.
You will probably have better luck with the tip of Master after commits from Friday, 11 MAR 2016. That would be roughly commits 0d4d5ab and 4c1cf7e. You can get the latest with:
git clone git://git.openssl.org/openssl.git
Or, use 1.0.2 stable releases.