Firebase function doesnt update, function returned undefined

You need to terminate a Cloud Function when all the asynchronous work is completed, see the doc. In the case of a background triggered Cloud Function (e.g. Cloud Firestore function onUpdate trigger) you must return the entire chain of promises returned by the asynchronous method calls.

exports.updateCourseName = functions.firestore
    .document('courses/{courseId}')
    .onUpdate((change, context) => {
      const newValue = change.after.data().name;
      const previousValue = change.before.data().name;
      const course1Id = context.params.courseId;
     
      if(newValue != previousValue){

        return admin   // <== See the return here
        .firestore().collection("set").where("course.id", "==", course1Id)
        .get()
        .then((querySnapshot) => {
          if (!querySnapshot.empty) {
         //       doc.data().author = newValue
                return querySnapshot.docs[0].course  // <== See the return here AND see below
                .update({author : newValue
                })
          } else  {
               return null;   // <== See here: we indicate to the CF that it can be cleaned up
          }
        })
        .catch((error) => {
            console.log("Error changing Author name in courses", error);
            return null;
        });
      } else  {
        return null;   // <== See here: we indicate to the CF that it can be cleaned up
      }
    });

HOWEVER, it is not clear to me what you want to do with:

querySnapshot.docs[0].course.update({author : newValue})

What is exactly course? The update() method is a method of a DocumentReference.

You probably want to do

querySnapshot.docs[0].ref.update({author : newValue})