Folding JSON strings to send in header variables - Mailgun

Question

I'm trying to use the JS Mailgun API to send emails. Have it working fine, until I throw template variables into 'h:X-Mailgun-Variables' like so, where jsonString is very large (17000+ characters):

const mailData = {
    from: 'Insights <[email protected]>',
    to: mailAddress,
    subject: `Insights: ${DAYS_OF_WEEK[date.getDay()]}, ${MONTHS[date.getMonth()]} ${ordinal_suffix_of(date.getDate())}  ${date.getFullYear()}`,
    template: "template1",
    'h:X-Mailgun-Variables': jsonString,
  };

Looking at the documentation here states the following:

Note The value of the “X-Mailgun-Variables” header must be valid JSON string, 
otherwise Mailgun won’t be able to parse it. If your X-Mailgun-Variables 
header exceeds 998 characters, you should use folding to spread the variables 
over multiple lines.

Referenced this post, which suggested I "fold" up the JSON by inserting CRLF characters at regular intervals. This led me here, which still does not work, though logging this does show regular line breaks and is compliant JSON:

const jsonString = JSON.stringify(templateVars).split('},').join('},\r \n');

Any insight into how to properly "fold" my JSON so I can use large template variables in my MailGun emails?

UPDATE:

As requested, adding my code. This works when data only has a few companies/posts, but when I have many companies each with many posts, I get the 400 error:

function dispatchEmails(data) {
  const DOMAIN = 'test.net';
  const mg = mailgun({apiKey: API_KEY, domain: DOMAIN});
  
  const templateVars = {
    date: theDate,
    previewText: 'preview',
    subject: 'subject',
    subhead: 'subhead',
    companies: data.companies.map(company => {
      return {
        url: company.url,
        totalParts: data.totalParts,
        currentPart: data.currentPart,
        companyData: {
          name: company.name,
          website: company.website,
          description: company.description
        },
        posts: _.map(company.news, item => {
          return {
            category: item.category,
            date: new Date(item.date),
            url: item.sourceUrl,
            title: item.title,
            source: item.publisherName,
            description: item.description,
          }
        })
      }
    })
  };

  const jsonString = JSON.stringify(templateVars).split('},').join('},\r \n');

  const mailData = {
    from: '[email protected]',
    to: '[email protected]',
    subject: 'subject',
    template: 'template',
    'h:X-Mailgun-Variables': jsonString
  };

  return mg.messages().send(mailData)
  .then(body => {
    return body;
  })
  .catch(err => {
    return {error: err};
  });
}

Have you taken a look at this? stackoverflow.com/questions/9779860/… — TechySharnav
– TechySharnav, Commented Mar 7, 2021 at 16:36
I have - tried those encoding methods, but still no dice. I know my json is formatted correctly because it goes through if I throw in a shorter version. But when the object is large, it errors out. So looking for a way to properly "fold" my json to conform to Mailgun's spec. I feel like I've implemented correctly now (see post), but still isn't working. — snn
– snn, Commented Mar 7, 2021 at 16:52
This may not always work because folding will have a requirement that your object's data is not large. You need to provide a sample json structure at least in your question — Tarun Lalwani
– Tarun Lalwani, Commented Mar 8, 2021 at 13:09
@TarunLalwani just updated with my code that generates JSON. I tried folding because the Mailgun documentation told me to use it if I was sending over long JSON, so assumed that would be the solution. But open to advice/suggestions! — snn
– snn, Commented Mar 8, 2021 at 19:31

Jeremy Moyers · Accepted Answer · 2022-05-03 14:58:33Z

I think your problem may be overall payload size rather than the string folding. The folding for strings exceeding 998 characters seems to be handled by the node.js client, possibly by the form-data package.

I ran the following test:

test_big_data (template created in mailgun through the UI)

<!DOCTYPE html>
<html>
    <body>
        <h2>An HTML template for testing large data sets.</h2>
        <ul>
            <li>{{param_name_0}}</li>
                ... other lis ...
            <li>{{param_name_999}}</li>
        </ul>
    </body>
</html>

send_email.js

const API_KEY = 'MY_KEY';
const DOMAIN = 'MY_DOMAIN.COM';

const formData = require('form-data');
const Mailgun = require('mailgun.js');

const mailgun = new Mailgun(formData);
const client = mailgun.client({ username: 'api', key: API_KEY });

const bigData = {};

for (let i = 0; i < 400; i++) {
    bigData[`param_name_${i}`] = `param_value_${i}`;
}

const dataString = JSON.stringify(bigData);
console.log(dataString.length);

const messageData = {
    from: 'Mailgun Sandbox <postmaster@DOMAIN>',
    to: 'test@MY_DOMAIN.COM',
    subject: 'Big Data Test',
    template: 'test_big_data',
    'h:X-Mailgun-Variables': dataString,
};

client.messages
    .create(DOMAIN, messageData)
    .then((res) => {
        console.log(res);
    })
    .catch((err) => {
        console.error(err);
    });

The length of the dataString in this case is 13781 characters and the email queues successfully.

However, if I bump the for loop condition to i < 1000 I get the following error when queueing the email:

[Error: Bad Request] {
  status: 400,
  details: '{"message":"Send options (parameters starting with o:, h:, or v:) are limited to 16 kB total"}\n'
}

When I asked Mailgun support about the folding warning form the documentation they pointed me to RFC 2822 section "3.2.3. Folding white space and comments". But like I said, I don't think folding is the issue here.

Cheers! https://datatracker.ietf.org/doc/html/rfc2822#page-11

JoshBerke · Accepted Answer · 2021-03-14 14:03:37Z

0

Just thinking outside the box but why pass that much data in an email header? I assume you have something on the receiving end which is going to parse the email headers. What if instead of sending them that data you send them a key that they can call back into an API on your end to get the data

answered Mar 14, 2021 at 14:03

JoshBerke

67.3k25 gold badges131 silver badges170 bronze badges

Collectives™ on Stack Overflow

Folding JSON strings to send in header variables - Mailgun

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related