Jawab:
(a.) (fog)(x) = 25x²+60x+38
(b.) (gof)(x) = 5x²-10x+22
(c.) (foh)(x) = x²-8x+18
(d.) (goh)(x) = 5x-8
(e.) g⁻¹(x) = ¹/₅ (x-7)
(f.)  h⁻¹(x) = x+3
(g.) (gog)⁻¹(x) = ¹/₂₅ (x-42)
(h.) (hoh)⁻¹(x) = x+6

Penjelasan:
Diketahui:
f(x) = x²-2x+3
g(x) = 5x+7
h(x) = x-3

Tentukan:
(a.) (fog)(x) = f(g(x))
= f(5x+7)
= (5x+7)²-2(5x+7)+3
= (5x)²+2(5x)(7)+7²-2(5x)-2(7)+3
= 25x²+70x+49-10x-14+3
= 25x²+60x+38
(fog)(x) = 25x²+60x+38

(b.) (gof)(x) = g(f(x))
= g(x²-2x+3)
= 5(x²-2x+3)+7
= 5(x²)-5(2x)+5(3)+7
= 5x²-10x+15+7
= 5x²-10x+22
(gof)(x) = 5x²-10x+22

(c.) (foh)(x) = f(h(x))
= f(x-3)
= (x-3)²-2(x-3)+3
= x²-2x(3)+3²-2x-(2(-3))+3
= x²-6x+9-2x+6+3
= x²-8x+18
(foh)(x) = x²-8x+18

(d.) (goh)(x) = g(h(x))
= g(x-3) = 5(x-3)+7
= 5x-5(3)+7
= 5x-15+7
= 5x-8
(goh)(x) = 5x-8

(e.) g⁻¹(x)
g(x) = 5x+7
y = 5x+7
x = 5y+7
5y = x - 7
y = ¹/₅ (x-7)
g⁻¹(x) = ¹/₅ (x-7)

(f.) h⁻¹(x)
h(x) = x-3
y = x-3
x = y-3
y = x+3
h⁻¹(x) = x+3

(g.) (gog)⁻¹(x)
Cari gog(x) dulu
g(g(x)) = g(5x+7)
= 5(5x+7)+7
= 5(5x)+5(7)+7
= 25x+35+7
= 25x+42
(gog)(x) = 25x+42
y = 25x+42
x = 25y+42
25y = x - 42
y = ¹/₂₅ (x-42)
(gog)⁻¹(x) = ¹/₂₅ (x-42)

(h.) (hoh)⁻¹(x)
Cari hoh(x) dulu
= h(h(x)) = h(x-3)
= (x-3)-3 = x-6
hoh(x) = x-6
y = x-6
x = y-6
y = x+6
(hoh)⁻¹(x) = x+6

(xcvi)