We recently upgraded from the Ruby 2.5 to the Ruby 2.7 Lambda runtime and we are now having issues with pdf generation. Of note, Ruby 2.7 is the first in the Ruby line using Amazon Linux2.

We're producing a fairly large pdf that took 5-10 seconds on the Ruby 2.5 runtime.

In troubleshooting, w've tried several different 14.2 distributions/layers:
* AWS Lambda - It appears to hang no matter how much memory I give it (timeouts up to 3.5 minutes). Because its hanging, I haven't been able to get much more information. The prince command itself runs if I just do `--version`
* Generic Linux x86_64 - it errors with a font error in ~ 3 seconds
```
msg|dbg||font request: serif
msg|wrn||Ensure fonts are available on the system or load them via a @font-face rule.
msg|wrn||For more information see:
msg|wrn||https://www.princexml.com/doc/help-install/#missing-glyphs-or-fonts
msg|err||Unable to find any available fonts.
fin|failure
```
* CentOS 7 64 bit - Iibxml2.so error almost instantly - `/opt/prince-20210624-centos7-x86_64/lib/prince/bin/prince: error while loading shared libraries: libxml2.so.2: cannot open shared object file: No such file or directory`

Do you have any recommendations on next steps in troubleshooting?

Can you successfully produce a small output document with the Prince package for AWS Lambda, for example a simple document like "<p>testing</p>" ?

Here is the verbose output of that html with the Lambda Layer:

prince: debug: init locking for OpenSSL
prince: debug: loading license: /opt/prince-engine/license/license.dat
prince: debug: loading /opt/prince-engine/license/license.dat because it is the main resource
prince: loading document: /opt/prince-engine/license/license.dat
prince: debug: loaded resource: /opt/prince-engine/license/license.dat
prince: debug: loaded resource: type: no
Fontconfig error: Cannot load default config file
prince: loading style sheet: /opt/prince-engine/style/fonts.css
prince: debug: loaded resource: /opt/prince-engine/style/fonts.css
prince: debug: loaded resource: type: no
prince: debug: enabling parallel downloads
prince: Loading document...
prince: loading HTML5 input: /tmp/test.html
prince: loading document: /tmp/test.html
prince: debug: error loading resource: can't open input file: No such file or directory
prince: /tmp/test.html: error: can't open input file: No such file or directory
prince: /tmp/test.html: error: could not load input file
prince: error: failed to load all input documents
prince: Finished: failure

Also of note, on the first round of tests, I had not removed the prince AWS lambda layer. Once I cleaned it up (just the generic linux prince layer), the generic linux errors with the below font error.

/opt/prince-20210624-linux-generic-x86_64/lib/prince/bin/prince: error while loading shared libraries: libfontconfig.so.1: cannot open shared object file: No such file or directory

The test above is failing to find /tmp/test.html, could you try running it again with the input file present?

The Prince package for AWS Lambda includes Fontconfig so it should work out of the box, whereas the Generic Linux package assumes that Fontconfig is already installed, so you will need to set it up yourself.

My bad, I grabbed the wrong one. that version had mismatched names ... Here is the version with the 14.2 AWS lambda running on the Ruby 2.7 runtime:

prince: debug: init locking for OpenSSL
prince: debug: loading license: /opt/prince-engine/license/license.dat
prince: debug: loading /opt/prince-engine/license/license.dat because it is the main resource
prince: loading document: /opt/prince-engine/license/license.dat
prince: debug: loaded resource: /opt/prince-engine/license/license.dat
prince: debug: loaded resource: type: no
Fontconfig error: Cannot load default config file
prince: loading style sheet: /opt/prince-engine/style/fonts.css
prince: debug: loaded resource: /opt/prince-engine/style/fonts.css
prince: debug: loaded resource: type: no
prince: debug: enabling parallel downloads
prince: Loading document...
prince: loading HTML5 input: /tmp/temp.html
prince: loading document: /tmp/temp.html
prince: debug: loaded resource: /tmp/temp.html
prince: debug: loaded resource: type: no
prince: debug: loaded document: /tmp/temp.html
prince: debug: sniffed doctype: XHTML
prince: Applying style sheets...
prince: loading style sheet: /opt/prince-engine/style/common.css
prince: debug: loaded resource: /opt/prince-engine/style/common.css
prince: debug: loaded resource: type: no
prince: loading style sheet: /opt/prince-engine/style/html.css
prince: debug: loaded resource: /opt/prince-engine/style/html.css
prince: debug: loaded resource: type: no
prince: loading style sheet: /opt/prince-engine/style/hyph.css
prince: debug: loaded resource: /opt/prince-engine/style/hyph.css
prince: debug: loaded resource: type: no
prince: loading style sheet: /opt/prince-engine/style/mathml.css
prince: debug: loaded resource: /opt/prince-engine/style/mathml.css
prince: debug: loaded resource: type: no
prince: loading style sheet: /opt/prince-engine/style/svg.css
prince: debug: loaded resource: /opt/prince-engine/style/svg.css
prince: debug: loaded resource: type: no
prince: Preparing document...
prince: Converting document...
prince: debug: pack
prince: debug: font request: serif
prince: debug: font scan: times new roman
prince: debug: font scan: times new roman, 0 matches
prince: debug: font scan: dejavu serif
prince: debug: font scan: dejavu serif, 0 matches
prince: debug: font scan: dejavu lgc serif
prince: debug: font scan: dejavu lgc serif, 0 matches
prince: debug: font scan: liberation serif
prince: debug: font scan: liberation serif, 0 matches
prince: debug: font scan: opensymbol
prince: debug: font scan: opensymbol, 0 matches
prince: debug: font scan: dejavu sans
prince: debug: font scan: dejavu sans, 0 matches
prince: debug: font scan: ar pl uming cn
prince: debug: font scan: ar pl uming cn, 0 matches
prince: debug: font scan: ar pl sungtil gb
prince: debug: font scan: ar pl sungtil gb, 0 matches
prince: debug: font scan: kochi mincho
prince: debug: font scan: kochi mincho, 0 matches
prince: debug: font scan: ipamincho
prince: debug: font scan: ipamincho, 0 matches
prince: debug: font scan: takaomincho
prince: debug: font scan: takaomincho, 0 matches
prince: debug: font scan: unbatang
prince: debug: font scan: unbatang, 0 matches
prince: debug: font scan: baekmuk batang
prince: debug: font scan: baekmuk batang, 0 matches
prince: debug: font scan: lohit devanagari
prince: debug: font scan: lohit devanagari, 0 matches
prince: debug: font scan: lohit bengali
prince: debug: font scan: lohit bengali, 0 matches
prince: debug: font scan: ani
prince: debug: font scan: ani, 0 matches
prince: debug: font scan: mukti narrow
prince: debug: font scan: mukti narrow, 0 matches
prince: debug: font scan: lohit punjabi
prince: debug: font scan: lohit punjabi, 0 matches
prince: debug: font scan: lohit gujarati
prince: debug: font scan: lohit gujarati, 0 matches
prince: debug: font scan: lohit tamil
prince: debug: font scan: lohit tamil, 0 matches
prince: debug: font scan: lohit telugu
prince: debug: font scan: lohit telugu, 0 matches
prince: debug: font scan: lohit kannada
prince: debug: font scan: lohit kannada, 0 matches
prince: debug: font scan: lohit malayalam
prince: debug: font scan: lohit malayalam, 0 matches
prince: debug: font scan: lohit oriya
prince: debug: font scan: lohit oriya, 0 matches
prince: debug: font scan: garuda
prince: debug: font scan: garuda, 0 matches
prince: debug: font scan: joypixels
prince: debug: font scan: joypixels, 0 matches
prince: debug: font scan: noto color emoji
prince: debug: font scan: noto color emoji, 0 matches
prince: warning: Ensure fonts are available on the system or load them via a @font-face rule.
prince: warning: For more information see:
prince: warning: https://www.princexml.com/doc/help-install/#missing-glyphs-or-fonts
prince: internal error: Unable to find any available fonts.

Hi,

Could you please try editing the "fontconfig/fonts.conf" file from the AWS Lambda package to look like this:

<fontconfig>
  <dir>/var/task/fonts/</dir>
  <dir>/opt/fonts</dir>
  <cachedir>/tmp/fonts-cache/</cachedir>
  <config></config>
</fontconfig>

Hello,

Same error:
(`cat /opt/etc/fontconfig/fonts.conf`)
I, [2021-10-19T13:55:17.489736 #9] INFO 4c1501f3-b8ce-4552-8d15-7008e8573198 -- : <fontconfig>
<dir>/var/task/fonts/</dir>
<dir>/opt/fonts</dir>
<cachedir>/tmp/fonts-cache/</cachedir>
<config></config>
</fontconfig>

I, [2021-10-19T13:55:17.491057 #9] INFO 4c1501f3-b8ce-4552-8d15-7008e8573198 -- :
prince: debug: init locking for OpenSSL
.......
prince: debug: font scan: noto color emoji, 0 matches
prince: warning: Ensure fonts are available on the system or load them via a @font-face rule.
prince: warning: For more information see:
prince: warning: https://www.princexml.com/doc/help-install/#missing-glyphs-or-fonts
prince: internal error: Unable to find any available fonts.

Thanks for that. I'm still unable to replicate your issue (using the Ruby 2.7 AWS Linux 2 Runtime).

In Ruby, are you able to try seeing if the Prince fallback fonts exist:

{ output: `ls -l /opt/prince-engine/fonts` }

It looks like it does:

I, [2021-10-19T21:12:24.207517 #9]  INFO e7cba65c-60e3-4f52-9361-2892ba932a9f -- : total 1
-rwxr-xr-x 1 root root 1000 Apr 23 04:47 prince-fallback.ttf

I stripped it down to a very basic function and I'm still seeing the error. Any chance you can share the layer (zip) you're using?

- Thanks

require 'json'

def lambda_handler(event:, context:)
    # TODO implement
    puts `echo "<p>testing</p>" >/tmp/temp.html`
    puts `/opt/prince-engine/bin/prince  --verbose --debug  /tmp/temp.html -o /tmp/temp.pdf`
    { statusCode: 200, body: JSON.generate('Hello from Lambda!') }
end

Ok, I think I know why I'm not seeing your error. My test was not using layers and so everything was in "/var/task".

I'll recreate my test using layers and will go from there.

Are you able to modify your "fontconfig/fonts.conf" to the following please, and then try to create another PDF please:

<fontconfig>
<dir>/var/task/fonts/</dir>
<dir>/opt/prince-engine/fonts</dir>
<cachedir>/tmp/fonts-cache/</cachedir>
<config></config>
</fontconfig>

Just an update: running Prince within a normal Lambda seems to work fine for me (both AWS Linux 1 and AWS Linux 2), but once I add Prince via a layer, I seem to be getting no output when giving it an input file (--version works file).

Once I pin down the issue, I'll let you know.

Thank you

I think I've found the issue...

When a normal Lambda is created, AWS unpacks the zip into "/var/task". If you unpack the Prince AWS zip file into your code directory before packaging it up, Prince should work out of the box. However, once you use Lambda Layers, each layer is instead unpacked into "/opt" with each layer clobbering any existing files from previous layers.

The "fontconfig/fonts.conf" file in the zip file on our website currently contains a single font directory ("/var/task/fonts"). The extra entry above (adding "/opt/fonts") should have fixed things for you, whereas the other attempt (adding /opt/prince-engine/fonts) would have only add our Prince Fallback font.

What I'll do on my end is add both "/opt/fonts" and "/opt/prince-engine/fonts" to the website zip file for the next version.

As for you to get it working, here's what I did to get Prince to produce a PDF:

First, I created the test HTML in the same directory as my Lambda, which when packaged up, should live in "/var/task/test.html":

  echo '<p>Testing</p>' > test.html

Next, I rebuilt the Prince zip to contain the "opt" directory (note to keep the directory structure the same i.e no subdirectory):

1) Unpacked the Prince zip file into a temporary directory
2) Added "/opt/fonts" to "fontconfig/fonts.conf"
3) Zipped up the contents back up
4) Uploaded the new zip file to S3

Next, published the new layer:

aws lambda publish-layer-version \
  --layer-name prince-layer \
  --content S3Bucket=<bucket>,S3Key=prince-14.2-aws-lambda-added-opt.zip \
  --compatible-runtimes ruby2.5 ruby2.7

Here's my test Ruby Lambda:

 require 'aws-sdk-lambda'
 $client = Aws::Lambda::Client.new()
 $client.get_account_settings()

 def lambda_handler(event:, context:)
   prince = `/opt/prince --debug -v /var/task/test.html -o /tmp/test.pdf 2>&1`

   if (File.exists?("/tmp/test.pdf"))
     file   = `base64 /tmp/test.pdf`
   else
     file   = "no file yet"
   end

   {
     os:    `cat /etc/os-release`,
     task:  `ls -l /var/task`,
     opt:   `ls -l /opt`,
     conf:  `cat /opt/etc/fontconfig/fonts.conf`,
     fonts: `ls -l /opt/fonts`,
     prince: prince,
     tmp:   `ls /tmp`,
     file:  file,
   }
 end

Next, publish the code and then modify the Lambda with the new layer:

  aws lambda list-layers 

  aws lambda update-function-configuration \
  --function-name <function-name> \
  --layers \
    arn:aws:lambda:<region>:<account>:layer:<function-name>:<function-layer-id-from-output-above> \
    arn:aws:lambda:<region>:<account>:layer:prince-layer:<prince-layer-id-from-output-above>

If all goes well, when your Lambda is run, the "file" entry in the output should be quite large which contains your PDF Base64 encoded.

I've confirmed that the above works with both AWS Linux 1 (Ruby2.5 runtime) and AWS Linux 2 (Ruby2.7 runtime).

Let me know if you have any issues following the above.

Edited 20 Oct 2021 by alfie

Thank you.

I also needed to switch to using

/opt/prince 

instead of

/opt/prince-engine/bin/prince 

. I assume it wasn't loading the configs correctly in AL2 when using the nested path. I'll port it to the full code base now and confirm.

Edited 20 Oct 2021 by ericstevens

We're now working with the full code base.

Thanks!

Excellent, glad it's all working for you